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I. INTRODUCTION 


A. GENERAL 

The search for better ways to do things in government is not new. 
However, in Naval ship acquisition, a major attempt at better decision- 
making is developing. 

Both weapons and systems for delivering them have gone through 
several revolutions in the years since the end of the Second World War. 
Nuclear bombs are now thousands of times more powerful than those 
dropped on Hiroshima and Nagasaki. Breakthroughs in missile tech- 
nology are threatening existing systems. Men in space have provided for 
military possibilities not thought of two decades ago. Keeping ahead in 
the technological race is not in itself a guarantee of security in these 
circumstances. It remains essential to incorporate developing tech- 
nology into operational hardware and to deploy and use this hardware 
with skill and intelligence. No amount of production skill or intelligent 
use can compensate for significant shortcomings in the development, test, 
and evaluation of the emerging hardware. These important first steps 
will be explored inthis thesis. | 

In parallel with the technological revolution in weapon systems in 
the past few decades, we have seen the costs of building and maintaining 
U. S. Naval ships multiply to astounding heights. The magnitude of these 


costs is emphasized by the recognition that overhaul and modernization of 
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computerized missile ships now is often more expensive than procure- 
ment costs of ships inthe past. Inthe fiscal-year 1975 Navy budget of 
$29.6 billion dollars, over $3.6 billion is allocated to the procurement 
of six classes of Navy vessels. The process of procuring ships is the 
longest, most complex and greatest in dollar volume, of any in the 
Defense Department. The cycle from concept to delivery of a new ship 
may require five to ten years, with technical improvements evolving 
the whole time. 

Weapon systems procurement functions are located at the highest 
organizational levels in the Services and involve concentrations of the 
top technical, procurement, test, and evaluation talent. This has been 
driven by aniterative process whereby the accelerating state of the art 
has driven the design methodology, management requirements, and sub- 
sequently the costs. Program management has been continually revised 
in an effort to keep stride with this process and ensure the best ships 
possible while keeping effective cost control. Thus the process des- 
cribed in this thesis consumes major portions of the resources of the 
Navy. 

Previous to World War II, the Navy did much of its own design and 
development and even some production. But after the war the Navy lost 
much of this in-house capability. Consequently, the Navy has of neces- 
sity become increasingly reliant upon industry for almost every aspect 
of weapon design, development, test, evaluation, and production. The 


Navy has shifted its thinking over the years from an absolute autocratic 
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control of this process inan internal environment, to a more tenuous 
and democratic committee control ina largely external contractor 
environment -- one that is much more inthe public eye. Unfortunately, 
the Navy hasn't made this transition smoothly. Within this context, one 
area which has not kept pace and is sorely in need of a quantum improve- 
ment is the development and management of test and evaluation programs. 
These should be structured to assist top management in making procure- 
ment decisions and to reduce the uncertainties surrounding these decisions. 
Test and evaluation of proposed new weapon systems is one of the 
Navy's key controls in the complex process of acquiring today's multi- 
billion dollar systems. Testing at decisive stages of development shows 
where problems exist and helps Navy managers to make better decision 
affecting future production and purchases of weapons than would other- 
wise be possible. The test and evaluation of emerging systems within 
itself involves uncertainty however. ''How much is enough?'’ must be 
answered for testing itself. It is apparent that many of the problems of 
managing a test and evaluation program have important economic aspects. 
The fundamental problem of how much to spend ona test and evalua- 
tion program is an allocation problem within the ship acquisition program. 
Economic theory tells us that we "should" continue to spend on test and 
evaluation until the marginal gain from the expenditures is just equal to 
the gain from expenditures elsewhere. But in the case of test and evalua- 


tion, this formula is peculiarly difficult to interpret or apply. The gain 
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is much more uncertain -- much harder to predict with accuracy -- than 
the gain from, say, an additional fighter defense squadron with aircraft 
of known performance. This Fe particularly true of developmental test 
and evaluation, where the product, ifany, will be mostly knowledge -- 
and knowledge usually far removed from its practical end use. Rather, 
its purpose is one of facilitating the evolution of the ship system itself. 
Rapid advances in electronics and computer technology have had a 
profound effect on the complexity of shipboard systems and the test and 
evaluation needed to support the procurement of these systems. There 
is a definite need in the Fleet for ever-increasing speed and accuracy in 
shipboard data processing to meet more sophisticated, high-speed threats. 
Asa result, the key weapon and sensor systems are utilizing digital inter- 
faces and processing to as great an extent as possible. The cost and 
complexity of the equipment and a movement toward human-engineered 
ships does not allow us to keep the luxury of back-up and casualty mode 
operation for many of these functions that we had formerly enjoyed. The 
demands for greater maintainability and reliability are also forcing in- 
creased digitalization. The result is a level of hardware and software 


integration never previously encountered ina ship system. 


B. THE PATROL FRIGATE PROGRAM 
The Patrol Frigate (PF 109) Class, for example, will have digital 
fire control systems inthe missile, gun and antisubmarine warfare 


areas, all linked to command and control within a centrally-located 





computer complex. To complicate matters further, at the time the PF 
program was being formulated, DOD policy regarding major weapon- 
systems development, acquisition and test was being drastically changed. 
In a Memorandum issued in the Spring of 1970, Deputy Secretary of 
Defense David Packard indicated that experience with the Total Package 
Procurement approach to development and acquisition of major systems 
was unsatisfactory; that it should be used with caution; and that a new 
policy would soon be established. The new policy took the form of DOD 
Instruction 5000. 1 which outlined the ''Fly-Before-Buy'"' approach. 
Essentially this new approach requires the services to develop and dem- 
onstrate the operability of a weapons system by actual test prior to the 
commitment of funds for mass-production of any major system. Progress 
in meeting this requirement is reviewed by the Defense Systems Acquisi- 
tion Review Council (DSARC) at discrete stages during development. 
DSARC determines whether the system is ready to advance to the next 
stage. It then makes recommendations to the Secretary of Defense con- 
cerning funding of the next stage and follow-on procurement. 

The very design of the PF evolved in parallel with emerging "'Fly- 
Before-Buy'"' policies within DOD. At the CNO/SECNAYV level it was 
soon realized that it was impractical from a cost and procurement 
schedule basis to adhere to the letter of the law of emerging policies. 
Tradeoffs were made such that adequate initial test and evaluation for 


subsystems is to be demonstrated prior to large-scale ship production. 
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Additionally, critical ship system integration is to be demonstrated at 
shore-based test sites. In effect then, we have swung from ''Total 
Package Procurement" to 'Fly-Before-Buy'"' with fewer paper solutions 
and more actual hardware testing, hardware proofing and prototyping at 
the system, subsystem, and component level. 

As early as the concept exploration phase of the PF procurement, 
it was realized that weapon selection and subsequent test and evaluation 
would be a controlling influence on the eventual size as wellas on the 
ship cost of the PF. The Ship Acquisition Project Manager (SHAPM), 
PMS 399, has identified the integration of the combat system as a critical 
task in the PF acquisition effort. In conjunction and compliance with 
emerging DOD test and evaluation requirements, the PF program has 
been structured specifically to provide for an extensive test program 
which should substantially reduce technical, cost, and schedule risks. 
The information obtained from an evaluation of combat-system test data 
will be a major factor in the final decisions on procurement of follow-ships 
at CEB, DSARC, and Congressional levels. 

Because of its critical nature, the combat system integration effort 
will be decoupled from the time-critical events of the shipyard and con- 
ducted at a land-based test site. The land-based test site will contain 
an installation duplicating, insofar as possible, the shipboard installation 
of all sensors, computing equipment, weapons and related subsystems. 


Interface equipment will be included. Major factors in achieving full 
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utilization of the integrated system which are to be demonstrated at the 
land-based test site are envisioned to be computer program debugging 
and physical, electrical and mechanical compatibility. Both of these are 


traditionally troublesome areas in complex ships. 


C. SCOPE OF THESIS 

The major objectives of this thesis were: to investigate and analyze 
emerging weapon system procurement, test, and evaluation policies 
within DOD and the Navy Department; to document one within the PF 
development process in view of these policies; and to gain insight into the 
use of a combat system land-based test site in implementing these pol- 


icies in the procurement cycle. 
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II. FROM ROCKS TO COMBAT SYSTEMS 


A. BACKGROUND 

Man, in his desire for a quicker means of destroying an enemy's 
capacity for waging war, has continually increased the complexity and 
destructive force of the weapons in his arsenal. From the era of the 
rock-throwing caveman, weaponry has now evolved to the era of nuclear- 
powered ships and sophisticated guided missiles. Early in the history 
of the Navy, the importance of forward-looking weapon system acquisition 
was recognized by the organizer of the Navy Gun Factory, John Adolphus 
Dahlgren, when he stated, "If the Navy be, indeed, the right arm of 
defense, her guns and ordnance... are the main sinews and arteries, 
the neglect of which soon <endee it feebled and palsied. ul 

Rightly called the "father of naval ordnance, '' Dahlgren revolutionized 
the science of ordnance with nis original work and set standards for other 
navies. In 1850 he urged construction of frigates armed entirely with 
heavy guns, anticipating the dreadnaughts of the next century. A blue- 
water sailor as well asa scientist, Admiral Dahlgren brilliantly com- 
manded a powerful naval force through two years of arduous wartime 
service during the Civil War, leaving behind him an example of produc- 


tive study and gallantry in action. 


IMemoirs of John A. Dahlgren, 1891, by Madelin V. Dahlgren. 
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Why the history? The point is that in the early days of weapon 
acquisition, test, evaluation and operational use, it was not all that 
unusual for Navy personnel to recognize a need for an improvement or 
new weapon, to design and manufacture the weapon within the Navy 
establishment, and for the conceivers and/or producers to take the 
weapon to sea for test and evaluation. Ifa deficiency in ordnance design 
was noted, it was often corrected on the spot by shipboard gunners. 

To a certain extent, a similar path was followed in the introduction 
of electronic devices to naval warships. Radio was developed as a result 
of the intense desires of men to exchange intelligence as rapidly as 
possible over long distances without physical interconnections. Until 
the present century, a ship's isolation was complete once she navigated 
out of port and sailed over the horizon. Of course, ships in company or 
in passing ''spoke'' to each other by voice hail, or signal flags by day, . 
and lanterns at night. But once at sea, orders or instructions from 
higher commands not in company could not be easily altered. For want 
of better communications, major battles have been fought after peace 
treaties were signed. 

The U. S. Navy, in keeping with its traditional scientific leadership, 
early recognized the impact of radio on naval operations. Only by such 
means of communication could far-scattered forces be effectively 
directed. Admiral Bradley A. Fiske, while a lieutenant, experimented 


with "electronic communication" on naval vessels in 1888, many years 
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before Marconi's successful application. During Fiske's investigations, 
he discovered the principles of degaussing ships as a protection against 
mines (used widely in World War II), and designed a system of radio 
control of torpedoes which forms the basis for the modern "radio" 
guided missile. Thus in the early days of electronics, Navy personnel 
often followed advanced concepts froma glimmering thought to useful 
hardware aboard ships. 

Prior to World War II, ordnance, electronics and communications 
were more or less separate and individual functions aboard ship. A new 
gun or electronic (radio) device could be developed, tested, evaluated, 
and placed in an operational status aboard ship in relative isolation. 
That is, weapons were weapons, and radios were radios, and there was | 
little marriage between them. 

During the 1930's the governments of Great Britain, the United 
States, and Germany fostered experiments to create a reliable detector 
that would locate airplanes not visible to the naked eye. The sound- 
detectors that had been used for this purpose had become almost use- 
less when the increasing speed of airplanes made it necessary for the 
gunners to "lead" their direction indicators by 20 degrees or more when 
firing. Although all three of the countries named were active in the 
development of radar, Great Britain led because it had the most to 
fear from air attacks. 

Radar was soon integrated with the fire control function of gunnery, 


since it provided a more sure method of determining ranges than optical 
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instruments and worked regardless of night, fog, or distance. The 
marriage of radar to gunnery in the area of fire control and the increas- 
ing swiftness of naval engagements, especially sea-air battles, brought 
into existence a new organization, the CIC or combat information center, 
on warships. The CIC filtered out the multitude of items of information 
that came in from the radar, spotting planes, sonar, and so on, reducing 
ler to manageable proportions to enable the commanding officer to 
make intelligent decors: 

Supersonic jets, greater threats from other surface units, and 
advancements in submarine warfare soon overran the man-machine 
interface present in CIC during much of World War II and the Korean 
War. Computers were introduced as an aid to commanders in their 
information gathering and decision-making processes. Application of 
the computer in naval weapons systems has advanced to the point where 
man-machine interface has been minimized to the point where in many 
instances human resources are no longer required. 

Throughout the course of history, the development of weapons has 
been extremely slow. It is said that until 1000 A.D., weapons hada 
useful life of about 400 years. Today, the useful life span of a weapon 
system may only be from 3 to 7 years before rendered obsolete by 
emerging technology or enemy advances. The design, development, 
test and evaluation phases of weapon-system acquisition have become 


increasingly important as the useful life span of weapons has decreased. 





It is the speed of development, test and evaluation which largely deter- 
mines the extent to which the system design incorporates the latest 


technical advances. 


B. WEAPON SYSTEM CONCEPTS 

The concept of a weapon system has come into being rather recently 
in naval history, evolving from ordnance, radio, radar and fire control 
to the point where a weapon system is made up of a number of unique, 
specialized components which must be integrated to achieve overall 
effectiveness. Ordnance Pamphlet (OP) 3000 defines a weapon system 
as "a collection of integrated components or subsystems which perform 
the interrelated functions necessary to render the desired effect on the 
enemy.'' Recent thinking has further integrated weapon systems into 
naval ships by dividing a ship into a containment system and a combat 
system. The containment system is viewed as the ship structure and 
related housekeeping systems, e.g., the hull, damage control, laundry 
and messes. The combat system is thought of as being the command 
and control, navigation, external communications, weapon and other 
supporting systems used directly for target surveillance, target recogni- 
tion, electronic countermeasures, weapon delivery, weapon handling 
and storage, and weapon safety. The complete combat system does 
not consist of the physical equipment alone, but also of technical manuals, 


training plans and equipment, spare parts, of an optimum employment 


21 Cex 





doctrine, and the human resources necessary to pursue objectives. 
Weapons system procurement thus includes concurrent development, 


test and evaluation of all of these items. 


Ze 





Tl. EVOLUTION OF TEST & EVALUATION CONCEPTS 


A. UP TO McNAMARA AND 1961 

From the perspective of Test and Evaluation, the weapon acquisition 
strategies which evolved within the Navy prior to the appointment of 
Secretary McNamara in 1961 suffered from a number of problems which 
paralleled those of planning and budgeting. For example, funds for 
adequate testing were often not available and test results were seldom 
available in time to influence the major program decisions which shaped 
the character and quantity of systems. Procurement of weapon systems 
was production orientated, with test and evaluation taking the back seat 
to hardware. Each major weapon subsystem was tested individually 
prior to shipboard deployment. The first time all interacting subsystems 
were tested as a single functioning system was after shipboard installation. 
Increasing system complexity and the requirement for the combat system 
to function under semi-automatic central control ina shipboard environ- 
ment was not mirrored in total system test and evaluation concepts. 

In conjunction with shortcomings in test and evaluation concepts, 
the government in the late 50's and early 60's regarded military procure- 
ment efforts as very risky and therefore used the cost-plus-fixed-fee 
type of contract for this work. During this period centralized govern- 
mental planning and direction of military procurement programs were 


lacking. As a result, many duplicative approaches were funded to 





meet poorly conceived military requirements, Programs were initiated 
without adequate evaluation of the time and cost required to complete 
them, and performance and reliability requirements were often left to 
the contractors to define after contract awards had been made. In this 
atmosphere, schedule slippages and cost overruns of 200% to 300% were 
typical, and many programs were cancelled before completion because 
of infeasibility, obsolescence, or excessive cost. 

Elements of each service sought funding for programs that would 
enhance their own military role. Funds usually being in short supply, 
the procuring service would often encourage the contractor to bid un- 
realistically low by revealing the amount of money available for the 
project. Inan effort to get hardware out of the factory and aboard ship, 
test and evaluation programs were often shortened or completely chopped 
from the procurement cycle. It remained for untrained sailors on board 
ship to discover that the weapon systems fell short of filling the require- 
ment for which they were originally procured. 

The nature of the contractors with whom the services were dealing 
also were going through changes. In years past, the Navy dealt with 
contractors who were actually producers. These were real technical 
people with responsibility but who were ona first name basis with Navy 
personnel. These people had been at sea and knew the bow from the 
stern ofa ship. The Navy could expect technical performance from 
men like this who felt a strong personal pride in any equipment they 


provided. 





During this time period, for a variety of complex reasons, most 
defense contractors were moving toward conglomerate corporations. 
The Navy no longer talked with owner-operators. They talked to cor- 
porate executives who often knew little about building ships and weapon 
systems but were interested in cash flow and return on investment. They 
were often lacking in detailed technical knowledge of weapons, having 
reached their management position through other channels, but were very 
good about discussing concepts such as schedules and costs. They may 
have known the cost impact of a proposed change, but they didn't know the 
other ramifications. | 

Since the contracts were the cost-plus-fixed-fee types and the con- 
tractor was assured of recovering his full cost, the contractor often was 
willing to bid well below the probable final cost in competing for the 
contract. Both the service and the contractor hoped that after the initial 
money was spent, the government would be so committed to the program 
that it would provide successively greater increments of funding rather 
than cancelit. A similar "liars contest'’ was conducted regarding the 
project schedule. If the government wanted the equipment by a certain 
date, the contractor was willing to commit himself to provide the equip- 
ment by that date, regardless of the chances of doing it, again often at 
the expense of test and evaluation, which was the final activity in the ~ 


schedule. 


25 \ 







Se SE ew a 

Ca —_ - a — 

a=. ao @= ea =a ae - 
—_— @ 4am 


- — = — 


B. McNAMARA'S POLICIES 

When Robert S. McNamara became Secretary of Defense, he launched 
a series of changes in the government's decision-making processes and 
in’ weapon-system contracting procedures which profoundly and rapidly 
affected the defense industry environment. He began by centralizing 
much of the authority to initiate programs in the Office of the Secretary 
of Defense. To avoid duplication of procurement in the services, he 
utilized the concept of Systems Analysis, which was to select among 
competing programs. He placed government budgeting ona five-year 
basis to permit appraisal of the total financial impact of major programs 
spanning a number of years. He required contractors to plan their 
activities thoroughly before the government approved the program 
funding. 

To dissuade contractors from accepting unrealistic contracts and to 
encourage efficient performance after award, McNamara curtailed the 
use of cost-plus-fixed-fee contracts. To motivate contractors to design 
weapon systems having low production costs, and to discourage them 
from bidding low on weapon system contracts in the hope of making it up 
in the subsequent sole-source production contracts he authorized the use 
of "Total Package Procurement"! contracts in which the development, 
test and evaluation, production, and support of weapon systems were 


competitively procured ina single, multi-year contract. 
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The Pentagon's new policy was bound to cause great difficulties for 
the defense industry, accustomed as it was to the old way of doing 
business. Its problems were complicated by the failure of the Defense 
Department to adapt completely to its changed role. DOD was weak, for 
example, in estimating program costs and in predicting how much test 
and evaluation was neces sary to reduce uncertainty in weapon system 
procurement to a manageable level. Policies established at high levels 
were not understood, implemented, or enforced at lower levels. The 
bureaucracy was doing a very poor job of defining the documentation 
necessary to implement the new procedures. 

Because the contractors were accustomed to telling their customers 
what they wanted to hear about socees dates, program cost, and ae 
uct performance, they found it almost impossible to bring themselves to 
challenge the government's preliminary plans. Yet this kind of confronta- 
tion was one of the major objectives of the contract definition phase of 
Total Package Procurement. 

Contractors, would, for example, consider the Pentagon's requested 
delivery dates as sacred, and tried somehow to schedule the design, 
development, test and evaluation, and production into the available time. 
They tended to make cost and technical appraisals by automatically 
accepting government requirements for the various elements of the 
system and program -- rather than by estimating each element's cost 


or technical performance individually, then summarizing them to 
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obtain an independent appraisal of total system performance and total 
program cost. 

As a result, the weapon system procurement plans were not much 
more realistic than those formerly submitted by contractors, when they 
simply told the services whatever would help them "'sell'' the programs 
further up the line. As in former years, when schedule and/or cost 
crunches arose, one of the first areas to be lopped off was the area of 
test and evaluation in an effort to get the hardware delivered. Paper 
studies were often substituted for actual test and evaluation of end 
product items. In short, contractors often signed fixed-price, total- 
package contracts at prices below the expected cost, containing risks 
that were not thoroughly appraised, and for which they lacked the 
management disciplines neces sary to perform the work in an efficient 


manner. 


C. GET WELL PROGRAMS 

Efforts to correct problems related to test and evaluation (T&E) 
were Started in the late Sixties. These efforts were greatly accelerated 
with the Report of the Blue Ribbon Defense Panel (BRDP) of July 1970, 
and various T&E measures taken during calendar 1971 to deal with the 
issues raised by the BRDP. 

1. The Blue Ribbon Defense Panel's Findings 

The following excerpt is quoted from the BRDP section on 

"Operational Testing and Evaluation, '' pages 88-91 of the report of 
1 July 1970. 
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Everyone seems to agree that Operational Testing and 
Evaluation (OT&E) is very important; however, there are significant 
differences of opinion as to what it encompasses, what its proper objec- 
tives are, and what organization and methods are necessary to accom- 
plish it most effectively. 


It has been customary to think of OT&E in terms of physical 
testing (under various designations such as operational suitability 
testing, employment testing, service testing, or field experimentation). 
It is essential to recognize that the primary goal of OT&E is operational 
evaluation, and that while operational testing is very important it is only 
one method of evaluation. To be effective, OT&E must be a total pro- 
cess, using all appropriate methods of evaluation, which spans the 
entire cycle of a system from initial requirement until it is phased out 
of the operational forces. If OT&E were limited to physical testing, it 
would lose much of its opportunity to contribute to decisions on whether 
to produce a system, and would seldom be able even to influence the 
system's characteristics and capabilities in any major way. 


Much OT&E does, however, involve physical testing and, 
therefor, it is important to distinguish between ''functional'' testing and 
"operation" testing. 


Functional testing (often engineering testing) ((termed ''Develop- 
ment Testing''in this Guide)) is done to determine how well various 
systems and materiel meet design and performance contractual specif- 
ications -- in other words, whether they meet technical requirements. 


By and large, functional testing in and for the Department of 
Defense appears to be well understood and faithfully executed. Serious 
policy deficiencies are not apparent, and such failures in functional 
testing as occur can be primarily attributed to lack of technical com- 
petence, oversight, or procedural breakdowns. Functional testing is 
not considered to be a major problem area. 


Operational testing, on the other hand, is done to determine to 
the extent possible whether such systems and material can meet opera- 
tional requirements. It must provide advance knowledge as to what 
their capabilities and limitations will be when they are subjected to the 
stresses of the environment for which they were designed (usually 
combat). Operational testing must take into account the interface with 
other systems and equipment, tactics and techniques, organizational 
arrangements, and the human skills and frailties of the eventual users. 
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There has been an increasing desire, particularly at OSD level, 
to use data from OT&E to assist in the decision-making process. Un- 
questionably, it would be extremely useful to replace or support critical 
assumptions and educated guesses with quantitative data obtained from 
realistic and relevant operational testing. 

Unfortunately, it has been almost impossible to obtain test 

results which are directly applicable to decisions or useful for analyses. 
Often test data do not exist. When they do, they frequently are derived 
from tests which were poorly designed or conducted under insufficiently 
controlled conditions to permit valid comparisons. It is especially 
difficult to obtain test data in time to assist in decision-making. Sig- 
nificant changes are essential if OT&E is to realize its potential for 
contributing to important decisions, particularly where the tests and 
the decisions must cross Service lines. 

The Blue Ribbon Defense Panel made three recommendations on 
improvement of T&E. While subsequent actions do not constitute rote 
implementation of the letter of specific recommendations, they do con- 
stitute substantial implementation of their spirit. BRDP recommenda- 
tions called for establishment ofa ''Defense Test Agency'! under the 
control of an Assistant Secretary of Defense for Test and Evaluation. 
The rationale of these recommendations was (1) to make sure that 
adequate testing was accomplished by an agency not under the control 
of the people who were attempting to "'sell'' the system being tested and 
(2) to provide an unfiltered channel for test-based information into the 
major program decisions made by the Secretary of Defense at DSARC I 
and DSARC II. A third recommendation called for a separate program 


category for T&E. This measure was designed to prevent the use of 


funds required for an adequate T&E to cover development cost overruns. 
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2. Mr. Packard and the DSARC 

One of the first things that Mr. David Packard of Hewlitt- 
Packard realized when he came into DOD was the necessity for clarify- 
ing the responsibilities of the Secretary of Defense and his office vis-a- 
vis those of the Service Secretaries and their staffs. He quickly decided 
that there was a need for a very simple modus operandi by which the 
OSD would establish policy and make decisions at appropriate points 
and the military services would carry them out. Once having done its 
part, the OSD would step back out of the way of the performing organiza- 
tion. Furthermore, the various OSD offices, suchas that of the Director 
of Defense Research and Engineering and the Assistant Secretaries for 
Installations and Logistics, Systems Analysis, and Comptroller, should 
lay out their spheres of responsibility and authority, subject, of course, 
to Mr. Packard's approval. 

In a Deputy Secretary of Defense Memorandum dated 28 May 
1970 entitled ''Policy Guidance on Major Weapon System Acquisition, "! 
Mr. Packard required the Services to develop and demonstrate the 
operability of a weapons system by actual test prior ie the commitment 
of funds for production of a major system. Progress in meeting this 
requirement is reviewed by the Defense Systems Acquisition Review 
Council (DSARC) at discrete stages during development. DSARC 
determines whether the system is ready to advance to the next stage, 


and then makes recommendations to the Secretary of Defense concerning 


3] { 





funding of the next serge) Generally these reviews are held prior to 
initiation of 1) easineeaine development, 2) prototyping, and 3) system 
production. 

When the council considers the first two of the three major 
decisions, the Director of Defense Research and Engineering chairs it. 
For the third decision, he serves as a member, while the Assistant 
Secretary of Defense for Installation and Logistics, who is responsible 
for production and development, becomes the chairman. At this point, 
the Director of Defense Research and Engineering is the seller and the 
Assistant Secretary of Defense for Installation and Logistics is the 
buyer. 

After the DSARC finishes its deliberations, the Deputy Sec- 
retary of Defense makes a decision which holds unless new evidence 
arises to change it. For example, usually the program goes ahead 
smoothly until it runs into some kind of cost, schedule, or system- 
performance problem. If any such difficulty causes the breaching of 
established thresholds (i.e., tolerances on critical program parameters 
agreed upon by the Secretary of Defense and the responsible Service 
Secretary), the program comes up for another examination by the 
Secretary of Defense. Barring that kind of situation, the programs 
are run by the Services and the Secretary of Defense stays out of them -- 
except to keep informed on their status through regular program 


reviews and Selected Acquisition Reports. 
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In his memo of 11 February 1971, "Conduct of Operational 
Test and Evaluation, '' Secretary Packard wrote: 


Although each Service now has a somewhat different way of 
organizing for operational test and evaluation, it is apparent to me 
that this function can best be performed by an agency which is separate 
and distinct from the developing command and which reports the results 
of its test and evaluation efforts directly to the Chief of the Service. 
Moreover, within the Service headquarters staff, there needs to be an 
office with a clear OT&E identification to provide staff assistance 
directly to the Service Chief and to provide a headquarters focal point 
for the independent OT&E field agency. Thus, at the completion of 
Operational Service Testing in the Army, OPEVAL in the Navy, and 
Cat III testing in the Air Force, I would expect that the respective 
Service Chiefs would have a clear picture of the operational suitability 
of a weapon system for Service use, to include its principal deficiencies 
and limitations and the corrective actions required prior to full-scale 
introduction into the force. Accordingly, each Service is requested to 
restructure its organization for OT&E along the lines specified above. 


As a second step, Iam establishing a Deputy Director for Test 
and Evaluation within ODDR&E with across the board responsibilities for 
OSD in test and evaluation matters. This office will review and approve 
test and evaluation plans prepared by the Services and will provide an 
assessment of results obtained. 


In his memo of 21 April 1971, Secretary Packard prescribed 
measures to ensure the flow of T&E information into major program 
decisions. 


The DCP format focuses attention on relevant technical and 
operational information through the identification of achievement mile- 
stones. The function of the test and evaluation milestones is to insure 
the availability of critical technical and operational data prior to each 
major Defense System Acquisition Review Council action. 


The newly-established Deputy Director, Defense Research 
and Engineering (Test and Evaluation) will receive from the Military 
Departments for review and comment, during the drafting of the initial 
DCP, a brief (1-4 pages) statement of the critical questions or issues 
which the development tests will address. The coordinated DCP for 
DSARC Milestone I (Program Decision) will contain a condensed 








version of the critical questions or issues as wellas the proposed 
schedule of the major development test milestones for addressing them. 


The DCP for DSARC Milestone II (Ratification Decision) will 
refine the critical questions or issues associated with development test 
and evaluation and will extend the process to include the critical ques- 
tions or issues to be addressed during initial OT&E and the test sched- 
ules therefor. At such time as the OT&E test plan is available, the 


Deputy Director will review it to insure that the critical questions or 
issues are adequately covered. 


Prior to the DSARC Milestone ll (Production Decision) the 
Military Departments will provide the Director of Defense Research 
and Engineering with an assessment of the test results interms of 
response to the initial questions or issues previously identified. The 
Deputy Director, Test and Evaluation, will review this assessment 
and provide an independent recommendation to the DSARC Milestone 
Hil me eting. 

3. Department of Defense Directive 5000. 1 

Department of Defense Directive (DODD) 5000.1, Acquisition 
of Major Defense Systems, the cornerstone of the Navy's comprehensive 
new acquisition policy, was the culmination of a concerted effort to 
identify the underlying causes of unsuccessful acquisition experiences 
and to prompt whatever improvements were deemed necessary. While 
DODD 5000. 1 addresses major weapon system programs, the manage- 
ment principles on which it is based are applicable to all programs. 
The directive formally establishes the DSARC. It takes a strong 
position on the decentralization of high-level controls in regard to the 
day-to-day aspects of acquisition, and it establishes the concept ofa 
powerful, less encumbered program manager. Finally, DODD 5000.1 


establishes a number of considerations for major weapon system 


acquisition. 





In the area of test and evaluation, paragraphs III. C.5 and 6 
apply directly and are as follows: 


5. Technical uncertainty shall be continually assessed. 
Progressive commitments of resources which incur program risk will 
be made only when confidence in program outcome is sufficiently high 
to warrant going ahead. Models, mock-ups and system hardware will 
be used to the greatest possible extent to increase confidence level. 


6. Test and evaluation shall commence as early as possible. 
A determination of operational suitability, including logistic support 
requirements, will be made prior to large-scale production commit- 
ments, making use of the most realistic test environment possible and 
the best representation of the future operational system available. 
The results of this operational testing will be evaluated and presented 
to the DSARC at the time of the production decision. 


4, More Guidance From Mr. Packard 


Secretary Packard's memo of 3 August 1971, ''Test and 


Evaluation in Systems Acquisition Process, '' amplified the guidance 


of his 11 February and 21 April memos in part as follows: 


The objective of the overall operational test and evaluation 
effort for any program is to aid in providing at major decision points 
in the acquisition and development process the best information possible 
at that point in time as to: the military utility of the prospective system; 
its expected operational effectiveness, operational suitability (including 
reliability, logistic, and training requirements); need for modification; 
and the organization, doctrine and tactics for system deployment. 
For programs intended for acquisition, phases of operational test and 
evaluation must be successfully executed in a timely manner to provide 
needed information as required. 


New acquisition programs requiring DSARC processing, or 
those which are currently in their early stages, will be so executed 
that an initial phase of operational test and evaluation will be accomp- 
lished prior to the major production decision to assist in estimating 
system operational effectiveness and suitability before that decision. 


a. This initial operational test and evaluation will be 
accomplished with operational personnel inas realistic an operating 





environment as possible and, where practical, will use pilot or early 
production items. 


The accomplishment of initial operational test and 
evaluation as a prerequisite for production decision does not obviate 
the necessity for completing operational test and evaluation when the 
_ production articles are introduced into the operating forces. 


5. From the Deputy DDR&E(T&E) 
In a memorandum for the Assistant Directors, the Deputy 
DDR&E(T&E) reviewed the three DepSecDef memos cited above and 
outlined the —— functions for his office: 


In summary then, the Deputy Director Test & Evaluation 
(DDT&E) (with the assistance of his staff) is to accomplish the following: 


1. Have across-the-board responsibilities for test and 
evaluation in the Department of Defense. Review test and evaluation 
policies and procedures of the DOD and the Military Departments and 
recommend changes as appropriate. 


2. Monitor closely test and evaluation programs conducted 
by the Services for DSARC programs and such other programs as he 
believes necessary throughout the entire testing cycle. Report to the 
DSARC and directly to DepSecDef at DSARC Milestones I and II his 
assessment as to the adequacy of the list of critical issues and prob- 
lems to be attacked by test and evaluation and the schedule of test 
milestones, and report at Milestone III to the DSARC and to the Dep- 
SecDef his independent recommendation. 


3. Insure timely OT&E to meet the intent of DepSecDef's 
direction described in 2a above. 


4. Request Service test plans and test results as may be 
required to accomplish (1), (2), and (3) above as early as such plans 
are developed by the Services and needed by DDT&E. 


5. Initiate and coordinate appropriate joint testing. 


6. Oversee the evaluation of foreign systems for possible 
DOD use. 





7. Administer for OSD its responsibility for the national 
and major Service ranges. 


8. Carry out such other T&E related activities as the SecDef, 
DepSecDef, DDR&E, or other DSARC member may direct. 


6. More from The Deputy 
LTGEN A. D. Starbird, the Deputy Director Defense Research 
and Engineering for Test and Evaluation, in an address to the 7th Naval 
Aviation Test and Evaluation Conference at NATC Patuxent River, Md., 
on 24 August 1971 commented on the evolution of test and evaluation 
concepts and emerging policies as follows: 


a. As we all know the Military Departments and the DOD have 
increasingly come into criticism in almost all phases of their activities. 
Unfortunately, test and evaluation has not escaped. In fact, on many 
Occasions it has come in for severe criticism. Each time a system in 
rather advanced stage of acquisition runs into trouble, there are critics 
who center on inadequate test and evaluation as a major culprit. 


b. It is interesting to me that the criticism of test and evalua- 
tion has come from many different quarters. On occasion, the press 
and the public have been critical. Congess in general has a sincere 
interest in seeing that we do adequate test and evaluation and their 
criticisms have been fairly frequent. The GAO has been critical, 
much more so recently than in earlier years. The President's 
Scientific Advisory Committee was asked to study out T&E and they 
emerged with certain criticisms. Many of you are probably familiar 
with the fact that the President's Blue Ribbon Defense Panel, which 
reported about a year ago, was quite critical of test and evaluation 
and particularly the quality of the operational test and evaluation 
accomplished by the Departments. 


c. Some of the critics have called for rather radical changes 
in how we do our T&E business. The Blue Ribbon Defense Panel, for 
example, recommended: appointment of an Assistant Secretary of 
Defense for Test and Evaluation to have a strong hand in approving 
Service test plans; and an OSD-level Test and Evaluation Agency to 
carry out the more critical DOD program tests. 
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We all realize that many of the criticisms leveled at our T&E 
are just not justified. Wein the DOD have done inthe past, and are 
doing now, much excellent testing. The results in the effectiveness of 
many of our complex systems show that our test and evaluation have 
been good. We devote a tremendous amount of effort of the best people 
and at high cost to test and evaluation. On balance, I believe that the 
Services are on the par with any other large industry, and perhaps 
ahead of most. 


But we have had faults and some of those persist. For these 
we are rightfully criticized. The faults in general fall in two categories. 

First is the availability of test results. Frequently the testing 
and its evaluation are completed long after critical decisions and major 
expenditure of funds have been made. We are well into production 
before we realize from tests that we will not reach the reliability goals 
specified. Weare well into deployment on occasion before we know 
that we cannot maintain adequately. The first valid criticism then is 
that much of our testing is not timely. 


Second, our key testing to determine operational effectiveness 
and suitability is often not accomplished until we are irrevocably 
committed at large dollar cost to some particular design. We do not 
accomplish early enough adequate operational test and evaluation. 
There are many definitions of operational testing but I think we all 
know what it means. It is testing of a product reasonably representa- 
tive of the final design, by operational units, in the type of environment 
the system will encounter when deployed, so as to establish its military 
worth and operational suitability. Included in that matter of operational 
suitability is whether its reliability requirements can be met and with- 
out undue maintenance. 


Secretary Laird and Secretary Packard have both been 
extremely interested in our test and evaluation from the time they took 
office with the Department of Defense. I remember two long sessions 
within the first week after they were sworn in when they met during 
most of one evening ona system for which I then had responsibility. 
Both asked many, many questions dealing with test organizations, 
test results and test plans. Both for many months have discussed 
with the Service Secretaries and the Military Chiefs what steps were 
necessary to accomplish improvements. Beginning early this calendar 
year, they issued a series of three instructions indicating what changes 
they desired to be made. The latest is the instruction by Secretary 
Packard of 3 August to which I earlier referred. Reading these col- 
lectively you can see that their objectives are really four in number. 
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First, and I believe this their primary objective, they desire 
to improve rather than revolutionize our test and evaluation approach. 
They want to build on the good that exists rather than to start again 
from scratch. 


They want to avoid "adding testing just for testing's sake." 
. They insist that everyone know the objective of their testing before 
embarking on their test program. 


They insist that the testing be timely, -- that it be adequate 
and scheduled for evaluation in time to contribute to the major decisions 
they must make. (I should point out that, for all important systems 
being acquired by the Services, they must approve moving a program 
from concept definition into advanced development, the subsequent 
movement into engineering development, and the third and more 
important decision to move into production. ) 


They insist too that the testing be realistic, that it be in 
sufficient detail and adequately evaluated to give realistic answers. 


7. T&Eand The Congress 
Congress, recognizing the importance of test and evaluation 
in the procurement of weapons systems, enacted in Public Law 92-156 
of November 17, 1971 a Section 506. A partial quote of that Section.as _. 
applies to test and evaluation follows: 


"Sec. 506. (a) Beginning with the calendar year 1972, the Secretary 
of Defense shall submit to the Congress each calendar year, at the 
same time the President submits the Budget... a written report 
regarding development and procurement schedules for each weapon 
system for which fund authorization is required... Beginning with the 
calendar year 1973, there shall be included in the report data on 
operational testing and evaluation for each such weapon system for 
which funds for procurement are requested (other than funds requested 
only for the procurement of units for operational testing and evaluation 
and/or long lead-time items)... 


'(b) A supplemental report shall be submitted to the 
Congress by the Secretary of Defense not less than thirty nor more 
than sixty days before the awarding of any contract or the exercising 
of any option in a contract for the procurement of any such weapon 





system (other than procurement of units for operational testing and 
evaluation and/or long lead-time items)... 


"(c) Any report required ... shall include detailed and 
summarized information with respect to each weapon system covered 
by such report, and shall specifically include, but shall not be limited 
tO -- oe. 

mt) ... the results of all operational testing and evaluation 
up to the time of the submission of the report, or, if operational testing 
and evaluation has not been conducted, a statement of the reasons there- 
for and the results of such other testing and evaluation as has been 
conducted. "' 
D,. IMPLEMENTATION DIRECTIVES AND INSTRUCTIONS 

1. SECNAVINST 5000. 1 

To implement DODD 5000.1, the Navy issued SECNAVINST 
5000. 1 and other flowdown directives which will be discussed later. 
Paragraph 6e of the basic instruction assigns COMOPTEVFOR 
(Commander, Operational Test and Evaluation Force) responsibility 
as an independent test agency for weapon system acquisition required 
T&E. It is interesting to note that enclosure (2) of the basic instruction 
canceled 28 previous instructions, of which three had T&E in their 
titles. 

Enclosure (3) to the basic instruction entitled ''Policy, 
Relationships and Responsibilities" contains the real meat of system 
acquisition direction within the Navy. For designated projects (Major 


Defense Systems such as those in Selected Acquisition Reports), the 


mode of operation for Project Managers is prescribed. The Project 
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Manager is directed to establish coordination with the DRDT&E 
(Director, Research, Development Test and Evaluation) to ensure 
adequate operational test planning and operational test and evaluation 
accomplishment. Section II of enclosure (3) indicates that test and 
evaluation is to be a consideration in RDT&E conceptual effort, full- 
scale development, and preproduction. Paragraph II.C.2.b. states 
that programs will not normally be proposed to the DSARC for SECDEF 
approval for production prior to completion of requisite operational 
test and evaluation. Deviations to prescribed T&E can be made only 


by or with the concurrence of the SECNAV. 


Section II, ''Program Considerations, '' in paragraph D 


further spells out T&E policy as follows: 


l. The wide variety of naval weapons dictates varying 
approaches to the conduct of test and evaluation; such effort shall be 
tailored to the needs and characterizations of each individual acquisi- | 
tion -- prime consideration being given to adequate operationally 
oriented testing. Normally, the following general sequence of events 
should prevail; (1) laboratory/contractor preliminary test and evalua- 
tion of breadboard of demonstration hardware during the conceptual 
effort, (2) contractor/development activity test and evaluation of sub- 
systems and/or full-scale demonstrator hardware during full-scale 
development, (3) technical test and evaluation conducted by the con- 
tractor with Navy/Marine Corps participation during pre-production/ 
production, (4) IOT&E (Initial Operational Test and Evaluation) by or 
with the active participation of Navy operational forces prior to the 
major production decision, (5) Navy/Marine Corps OPEVAL (Opera- 
tional Test and Evaluation) prior to approval for service use and 
inventory acceptance (except for ships), (6) Navy/Marine Corps 
follow-on test and evaluation, and (7) the conduct of tests and evalua- 
tions by the Board of Inspection and Survey (BIS) and recommendation 
to the Chief of Naval Operations for service acceptability. Test and 
evaluation effort shall be effectively correlated with previously out- 
lined requirements concerning approval of material for service uSe. 





Specifically, the procedural aspects of requirements determination, 
research and development, manufacture of service test model(s), tech- 
nical evaluation, initial operational test and evaluation, full operational 
evaluation, and approval for service use shall be correlated. 


2. New acquisition includes conversions, major modifications 
and modernizations. Adequate test and evaluation of these is also 
MeQuired .«s0 «+ 

3. For new acquisitions ... the principles of early operational 
test and evaluation before production decision, participation by the 
Marine Corps or COMOPTEVFOR shall apply. The purpose of IOT&E 
and other tests is to provide assessments and recommendations by an 
activity independent from the development activity concerning the future 
operational suitability of systems under development. 

4. The objective of the overall operational test and evaluation 
effort for any program is to aid in providing, at major decision points in 
the development and acquisition process, the best information possible 
at that point in time as to: the military utility of the prospective system; 
its expected operational effectiveness, operational suitability (including 
reliability, maintainability, simplicity, logistic, and training require- 
ments); need for modifications; and the organization, doctrine and 
tactics for system deployment... 

2. Congressional Concern 

Concurrent with the issuance of DODD 5000. 1 and the Navy's 
implementing instruction SECNAV 5000.1, there was increasing Con- 
gressional concern over testing of major acquisitions. Because of the 
major problems encountered with weapons programs asa result of 
inadequate testing, Congressional committees increasingly qualified 
the authorization of resources assigned to programs until comprehen- 
sive testing had been completed. Following are some examples from 


the fiscal year 1971 and 1972 authorizing appropriations to illustrate 


this increased interest by the Congress. The restraints applied most 
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often occur during requests by the services to place a major weapon 
in full-scale production. 
In Public Law 91-441, dated October 7, 1970, the Congress 


stated that: 

Of the total amount authorized to be appropriated by this Act for the 
procurement of the F-111 aircraft, $283,000,000, of such amount may 
not be obligated or expended for the procurement of such aircraft until 
and unless the Secretary of Defense has (1) determined that the F-111 
aircraft has been subjected to and successfully completed a comprehen- 
sive structural integrity test program... 


In House Report 92-232 on authorization appropriations for 
fiscal year 1972, SRAM and MAVERICK were discussed. 


The AGM-69A SRAM is an air launched air-to-surface missile for 
planned use on the B-52G/H and FB-111 aircraft. The SRAM missile 
is equipped with a nuclear warhead designed to attack targets defended 
by sophisticated defense systems. The fiscal year 1972 program pro- 
vides for proceeding to full-scale production considering completion of 
testing. 


MAVERICK is an air-to-surface Air Force missile, electro-optically 
guided for use against small hard targets such as tanks and bunkers. 
Last year the Congress, on the recommendation of the Committee on 
Armed Services, denied requested procurement funds for MAVERICK 
and directed that the program be continued in research and development 
to avoid concurrency and to allow more reliable development and test 
results prior to initiating procurement. 


3. T&Eand the GAO . 
At the request of the Honorable Carl Albert, Speaker of the 
House of Representatives, the General Accounting Office (GAO) made 
a review of the importance of testing and evaluation by the Department 


of Defense in the acquisition process for major weapon systems. 


ape Importance of Testing and Evaluation in the Acquisition Process 
for Major Weapons Systems, The Comptroller General of the United 
States, 7 August 1972. 
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Their report indicates that the benefit of testing is accomplished 
through properly assessing risks and delivering test results to the 
decision-maker at key points in the acquisition cycle when final de- 
cisions must be made. Testing at decisive stages of development was 
found to show where problems exist and to help military managers 
make sound decisions affecting future production and purchase of 
weapons than would otherwise be possible. 

The GAO reviewed 13 weapon systems with estimated fore 
costs of more than $46 billion. They included such weapons as the 
Army's improved HAWK missile, the Navy's DE-1052 (destroyer 
escort), and the Air Force's F-15 aircraft. On the basis of its ob- 
servations of the pattern of testing performance, the GAO concluded 
that: 


~-Practices used to establish objectives for testing generally were 
adequate. 


--Most weapon systems did not have adequate plans for conducting 
fests. 


--Testing and evaluation for most weapon systems was not accomplished 
ina timely manner. 


--Most test reports were adequate, but their value was diminished due 
to inadequate test planning and actual testing. Some reporting im- 
provements could be made. 

--Complete and valid test and evaluation data was not available prior 
to those times in the acquisition cycle when decisions had to be 
made. 


Based on DOD's listing of five phases in the acquisition 


process of weapon systems (1 concept formulation, 2 validation and 
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ratification, 3 development, 4 production, and 5 deployment) the 
GAO formulated a model (Appendix A) to depict the role of testing in 
the acquisition cycle. The purpose of GAO's model was to reinforce 
and emphasize certain ideal concepts of testing and evaluation in the 
acquisition process, as follows: 


-~- Testing and evaluation is an important ingredient throughout the 
acquisition process. 


--The sequential nature of engineering; acceptance; and, to some 
degree, operational suitability testing and evaluation. 


--The responsibility for the success of testing and evaluation through- 
out the acquisition process lies with the developer, the user, and 
the contractor in different degrees and at different times. 

-~-To provide a means for DOD and weapon systems program managers 
to evaluate the testing and evaluation process in their respective 
programs. 

4, lLaird's Viewpoint 
In his final report to the House Armed Service Committee on _ _ 

January 8, 1973, Secretary Melvin Laird indicated that with the help of 

David Packard, many changes had been made in the weapons system 

acquisition process but that it would take some years before the im- 

provements would be fully validated. He indicated that bankrupt prac- 

tices such as total package procurement and an indiscriminate use of 
concurrency between development and production had been eliminated. 

His substitutes included ''test before you fly'' and "fly before you buy" 


procedures, and the widespread use of contract milestones and 


prototyping. 
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5. DODD 5000.3 
In January 1973, the several earlier T&E memoranda were 
superseded by DOD Directive 5000.3 entitled ''Test and Evaluation." 

It was in somewhat more detail and tightened to a degree the earlier 

memoranda instructions: 

--It stated certain principles that should apply to all acquisition pro- 
grams. T&E would start as early as possible and be conducted in 
phases so as to eliminate risks early. Acquisition schedules must 
be keyed to accomplishing T&E milestones before major added 
resource commitments were made. FPrior to contracting for major 
production, there would be completed: sufficient development test- 
ing (DT&E) to insure that all design problems were identified and 
solutions were in hand; and sufficient operational testing (OT&E) to 
provide a valid estimate of operational effectiveness and suitability. 

--In each Service, OT&E would be the responsibility of a field agency 
independent of the developing command. Typical military operating / 
maintaining personnel and units were to carry out the test, -- not 
highly trained development technicians. Such operational testing 
was to be conducted ina simulated operational environment where 
the stresses of real combat operations (such as fatigue and counter- 
measures) were applied to the degree possible. 

--Operational tests preferably were to be separate from development 


tests. However, an early phase of operational testing could be 
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combined with a development test providing the combined test was 
participated in and accepted by both the developer and the independent 
operational test agency as satisfying their individual requirements. 
Pilot production items were to be used wherever possible for the 
development and operational testing completed before commitment 

to major production. However, final prototypes could be employed 
where these prototypes were truly like the expected production items. 

--For T&E of ships of a major class it was recognized that because 
of the long design, engineering, and construction period, completion 
of T&E prior to the decision to proceed with follow ships would not 
in all likelihood be possible. ‘In lieu thereof, successive phases of 
DT&E and OT&E will be accomplished as early as practicable at test 
installations ... to minimize the need for modification to follow 
ships.'' Use of prototypes to achieve T&E prior to the first major _ 
production decision on follow ships is advocated. 

--The directive included an important new provision with respect to 
waiving satisfactory completion of required T&E. For each major 
acquisition program, the describing Development Concept Paper 
(DCP) sets forth such required T&E and only the Secretary of 
Defense can waive its accomplishment. 

6. OPNAVINST 3660. 8 
OPNAV INSTRUCTION 3660.8 of 22 January 1973 established 


guidance and policy for T&E of Navy systems and equipments in 
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accordance with the requirements of SECNAVINST 5000. 1 and DODD 
5000.3. The instruction reemphasizes that Navy weapon system ac- 
quisitions shall provide for T&E programs to (1) permit a clear deter- 
mination of service acceptability and suitability by competent authority; 
(2) support the development activities in evolving a design which meets 
specifications and service requirements; (3) support the Secretary of 
Defense review and decision process. 

It recognized that conventional ship programs may be large, 
complex and prolonged evolutions. Consequently, development T&E 
required ona specified subsystem or groups of subsystems may be 
conducted on board other class ships or at land-based test sites. In 
the case of a new hull or propulsion designs prototypes, or land-based 
test sites, construction is normally undertaken to verify laboratory 
DT&E. 

While previous instructions and directives addressed T&E in 
general terms, 3960.8 defines T&E specifically and discusses T&E of 
weapons systems in the following areas: developmental testing, opera- 
tional testing, and acceptance trials. The following pertinent definitions 
are made: 

TEST: Any program or procedure which is designed to obtain, verify, 
or provide data for the evaluation of; research and development; pro- 
gress in accomplishing development objectives; or performance and 


operational capability of system, subsystems, components, and equip- 
mene 1cems. 





DEVELOPMENTAL TEST AND EVALUATION (DT&E): Test and 
evaluation performed by or for the developing agency which emphasizes 
the technological and engineering aspects of the system, subsystem, or 
equipment items. Normally carried out under strictly controlled 
conditions. 


OPERATIONAL TEST AND EVALUATION (OT&E): Tests and evalua- 
tions participated in or performed by operational personnel focusing on 
operational effectiveness and suitability (including reliability, compat- 
ibility, interoperability, maintainability, and supportability). It also 
includes the development of optimum operational tactics for systems 
and equipment being developed for service use. 


OT&E is further broken down into the following areas: 


INITIAL OPERATIONAL TEST AND EVALUATION (IOT&E): That 
T&E accomplished prior to the DSARC Milestone III or comparable 
CNO or CHNAVMAT first major production decision point to permit 
assessment of the operational effectiveness and suitability of a weapon 
system. 


FOLLOW-ON OPERATIONAL TEST AND EVALUATION (FOT&E): 
The continuing operational test and evaluation of a weapon system con- 
ducted in an operational environment by operational personnel using 
production systems for the purpose of verifying system performance; 
validating correction of deficiencies previously identified; and refining 
tactical employment doctrine and requirements for personnel and 
training. FOT&E may be initiated using pilot or pre-production sys- 
tems which most closely resemble the production units until the latter 
items are available. 


A comprehensive set of definitions of terms and phrases 
commonly used in T&E, checklists for T&E in support of milestones 
I through III of the DSARC process, and a diagram of the acquisition 
process are provided as enclosures. T&E responsibilities and 
relationships which have evolved are also discussed which will be the 


subject of the following section of this thesis. 





7. NAVMAT NOTICE 3960 

NAVMAT NOTICE 3960 of 1 February 1973, ''Test and Evalua- 
tion Policy for Major Surface Combatant Ships Project, " assigned the 
Major Surface Combatant Ships Project Manager, PM 18, responsibility 
for T&E effort within the Naval Material Command relating to major 
surface combatant ships. The ship classes include DLGN 36, DLGN 38, 
DG, DD 963, LHA, PF, SCS, and CVAN 68. 

The Notice recognized that inthe past, ship acquisition T&E 
philosophy had been tailored to the needs and production approach of 
each individual ship program. In light of recent DOD policy for major 
weapon acquisitions establishing more standard criteria for the approval 
of programs (DCP/DSARC), a need for adoption of more uniform T&E 
procedures was recognized. Guidelines for test planning are provided 
in conjunction with the PM-18 functional organization. Test processes 
and relationships in test design, development and execution are 
provided. 

The requirement for a Test and Evaluation Master Plan (TEMP) 
as initial policy in T&E for major surface-combatant acquisition pro- 
grams was laid on. The TEMP is to be keyed to acquisition milestones 
and should cover all T&E requirements through the production phase 
including follow-on operational testing. It should include development 
of the maintenance and ship-overhaul baseline test program for the 


equipment and system-level Planned Maintenance System (PMS). 
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It directed that test planning should be total-ship oriented to assure 
adequate coverage of all work breakdown structure groups and to 
assure that all systems are properly installed and completely operable, 
before a ship is presented for Board of Inspection and Survey (BIS) 
trials. Additionally, a Total Ship Test Director (TSTD) responsible 
for planning and execution of the T&E program is required for each 
program. 
8. NAVMATINST 3960. 6 

NAVMAT INSTRUCTION 3960.6 of 24 August 1973, ''Planning 
and Implementation of Tests and Evaluations of New Weapon Systems, " 
provides further guidance and direction for preparing the TEMP. 
Under this instruction, the TEMP is considered to be a management 
document and serves as a foundation for the more detailed DT&E and 
IOT&KE plans. Its basic elements are objectives, issues, responsibil- 
ities, interfaces and schedules. It should be viewed as a planning 
document rather than a control document. 

9. NAVMATINST 3960. XX 

NAVMATINST 3960. XX, a draft instruction of 21 December 
1973, is presently under review within NAVMAT. The instruction, 
"Test and Evaluation of Ship Acquisition, '' has been prepared to 
formalize NAVMAT policies and procedures for ship acquisition T&E 
in order not only to respond to DOD and OPNAV requirements but 
also to provide coordination and standardized approach to the develop- 


ment of test requirements and the supporting test documentation. 
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The Ship Acquisition Project Manager (SHAPM) is assigned 
the responsibility for developing from appropriate OPNAV design 
requirements and his own risk analyses the test requirements neces- 
sary to reduce ship acquisition risks. 

Enclosure (1) to this instruction, ‘Management Plan for Total 
Ship Test Program for Ship Acquisition, '' prescribes the basic manage- 
ment structure and procedures to be followed for developing test 
documentation that will satisfy the objectives of ship acquisition T&E 
and will also support the total ship test program (TSTP) for active 
fleet ships. This enclosure is a comprehensive document that attempts 
to bring together within a single cover T&E organization and responsibil- 
ities, test documentation requirements, and development ofa ship 
construction integrated test package. Itis against this background 
that the proposed and on-going T&E efforts of the Patrol Frigate 


~~ 


project will be compared and analyzed in this thesis. 





IV. SOME PROTOTYPING CONSIDERATIONS 


A, GENERAL 

Navy prototyping is still evolving in the fly-before-buy era. It is 
based primarily on past experience with prototyping and Total Package 
Procurement. It is also influenced by the puocedural and standardiza- 
tion guidance received ona continuing basis from the DOD as developed 
in the previous chapter. The literature concerning weapon system 
acquisition, DOD and Navy Instruction, and various memoranda are 
filled with positive statements concerning the use and advantages of 
prototypes. These statements take the form of: 
David Packard, ''We want to put more reliance on hardware and less 
On paper studies in advanced development. "' 
Dr. John S. Foster, Jr., ''We desire to emphasize the use of 
prototypes. '' 
Barry J. Shillito, ''Under prototype development, we will evaluate 
hardware rather than paper promises in choosing a contractor.'! 
CAPT E. J. Otth, SHAPM for the Patrol Frigate Acquisition, "We 
expect to use a land-based test site for the combat system as wellas 
the propulsion system to enable us to verify integration and perform- 
ance of these systems in an environment more controlled and less 


costly than aboard ship." 
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Early in 1974, discussions with Mr. Matthew T. Reynolds, 
NAVMAT's point of contact in PM 18 for all T&E matters relating to 
major surface ships, indicated that trends in prototyping of combat 
systems in new ship-class acquisitions had reached the point where 
prototyping was essentially mandatory. At least at this point of time 
in acquisition philosophy, prototyping is an '"'in'' concept. Therefore, 
what prototyping is and what expected gains and costs are incurred in 
its use are of considerable interest. 

Conventional wisdom has it that a prototype is a redundant and 
often costly adjunct to the design and development process. This thesis 
considers the proposition that under given circumstances the most 
effective way of resolving some of the uncertainties of combat system 
development is by building and testing a prototype system at a land- 
based test site. Underlying the discussion is the premise that no single 
specified pattern of behavior can adequately provide for all of the con- 
tingencies that will arise in the course of combat system development. 
Additionally, the premise that cost and the long lead and production 
times involved in ship construction do not permit the prototyping of 
entire ships prior to production decision points is accepted without 
argument or further.comment. The combat system portion of the 
total ship is broken out as a prime candidate for prototyping in contrast 
to the relatively fixed containment system which houses and supports 


the combat system. 
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B. CONCEPTS 

To many people, the word ''prototype'' evokes images of the 1930's 
and the 1940's, suggesting the angularity ofa biplane, the quaintness 
of goggles, and the hum of electronic equipment patched together with 
tape and spliced wires. But what is being discussed here is some- 
thing quite different; a vehicle for lessening technological uncertainty. 
It might be more appropriate to call that vehicle a definition-phase test 
combat system or perhaps a demonstrator combat system, rather than 
a combat-system land-based test site, if only to deter prejudgement 
concerning the utility of using a land system to emulate shipboard 
systems. But prudence suggests merely defining the term as it is to 
be used here and leaving language reform for another occasion. 

There are many possible themes in any consideration of the use 
of prototypes in development. The hypothesis from which this thesis 
reasons is that: (1) establishing the detailed configuration of a new 
combat system requires many decisions, each difficult, and each 
dependent on the resolution -- or avoidance -- of uncertainty; (2) the 
elemental substance of T&E is error and uncertainty, and delay in the 
detection and correction of error or oversight is a principal cause of 
inefficiency or ineffectiveness of development; (3) the early resolution 
of uncertainties in technology, objectives, schedules, and costs is 
vital, there are various ways of resolving such uncertainties, anda 


technique of resolution most appropriate to the problem at hand should 
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be chosen; and (4) in some circumstances, which can be defined, the 
most desirable way of resolving technological Hine RNS in ship 
acquisition programs may be to build and test a prototype combat 
system. 

There are many definitions of "prototype'' and little profit in laying 
them end to end for scrutiny. Basically this thesis will considera 
prototype combat system to be a hardware-software combination 
approximating, in full scale, the main features ofa prospective opera- 
tional combat system. lLand-based tests must yield information that 
will permit the timely identification and resolution of technological 
uncertainties and ultimately support a sound decision about the procure- 
ment of production systems. Additionally, -the.prototype may be used 
to suggest ways the design, structure and development of the combat 
system can best accommodate the shipbuilder's requirements for ___ 
detailing the design of cabling and ducting, assess maintenance require- 
ments, and be useful throughout the life-cycle of the production system 
as a test and configuration bed. Owing to the nature ofa prototype, it 
is not very likely that all prospective subsystems will be available as 


early as selected hardware items, and some simulations may have to 


be used in place of other items. 


cee VWoeNetO PROTOTYPE 
The following figure (p. 57) presents some concepts for considera- 


tion in prototyping decisions. Seen thus, a prototype is a device to 
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moderate some of the uncertainty that characterizes the decision 
process. Because a prototype is a full scale, semi-operable representa- 
tion of what is intended to be in time an operational combat system, 
decisions buttressed by experience with prototypes tend to be founded 
on less tenuous technological grounds than those based on abstract 
analysis and derived from assumptions. Reliance on prototypes appears 
to be advantageous for four general classes of development decision. 
(1) When there is uncertainty about the advisability of producing a 
given combat system, testing a prototype will provide reliable informa- 
tion about the attributes of the combat system and will reduce the 
quantity and importance of misgivings. (2) When there is uncertainty 
about which of two similar hardware/software combinations to produce, 
testing subsystems built by different contractors will ease the decision. 
In the same vein, if cost overruns or technical inadequacy of subsys- _ 
tems causes fall back to other equipment, a prototype lends a suitable 
means for incorporating these emerging configuration changes. 
(3) When there is uncertainty about which combat system to build, 
testing prototypes, liberally supported with realistic simulations, will 
provide hard information on which a choice can be based. (4) When 
there is uncertainty about the technological feasibility of a configura- 
tion, prototype testing can assist in resolving the main issues. 

The matter of whether a prototype program is inherently more or 


less costly than a program that bypasses the prototype option can be 
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treated, for the moment, as incidental. It is apparent that cost and 
schedule estimates derived in part from experience with prototype 
combat systems will tend to be somewhat less uncertain than estimates 


based solely on analytical predictions of development progress. From 
the fact that specific uncertainties of technology, configuration, and 
probable mission responsiveness will be moderated by the very exist- 
ence of a prototype, it may be adduced that a prototype strategy can be 
used as a means of preserving a production option -- putting off a final 
decision until requirements of technology are better understood while 
concurrently working toward that understanding. 

There is no inherent reason why "Total Package Procurement, " 
should not include a prototype phase if the ''total package'' decision 
specified quantities and schedules but made configuration definition a 
function of an evaluation process following prototype testing. Or, for | 
that matter, ‘'total package'' could be interpreted to mean all develop- 
ment and production activity following a definition phase that provided 
for building and testing prototypes. Within current thinking however, 

a prototype provides a vehicle to "test before flying, '' and ‘'flying 
before buying. " 

A prototype should be built in the expectation of change, and the 
expectation of change is its only substantial justification. The objec- 
tive of building a prototype is to discover what changes are necessary -- 


what decisions must be made -- before a design is committed to 
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production, or to discover if any number of changes will end ina 
desirable production combat system. Sucha characterization does 
not invalidate the use of prototypes to aid in resolving mission uncer- 
tainty or to aid in equipment/contractor selection or to contribute toa 
decision on which (rather than what) to produce, but it emphasizes the 
contribution of a prototype to the diminution of uncertainty, whatever 
the nature of that uncertainty. 

From such assumptions arise two principles for the conduct of 
programs based onthe use of prototypes: (1) controlled investment -- 
that is, obtaining an informational return at an expenditure commen- 
surate with the worth of the information, and (2) defining and limiting 
uncertainty; which is to say, having the principal features of a combat 


system reasonably well defined before undertaking construction. Al- 


-- though planners may squirm at the thought, there is no obvious reason., 


for specifying the detailed operational assignment of a combat system 
while a prototype is being built and tested; the question of whether it 
should have one or two digital computers cannot be put off in the same 
way. Flexibility however, must be built into the system to enable 
modification to reasonably expected equipment and/or threats in the 
planning future. 

A prototype combat system should not be expected to resolve 
uncertainties that are peripheral to the main purpose of building it, 


nor should peripheral uncertainties be permitted to dominate a 





prototype program. Technical problems that can readily be identified 
and solved by analysis and discrete off-site testing should not be left 
to a prototype test inquiry. Nor should very great technical uncertain- 
ties be left for resolution by means of a prototype combat system. 
Basic technological feasibility is not an issue that can be satisfactorily 
resolved at the prototype level; feasibility demonstrations cannot be 
put off until operational requirements begin to hinge on their success. 
A prototype is not a specific vehicle for uncovering "unks-unks" nor 
for resolving basic policy differences between contributing factions. 
When the time or urgency of a requirement is in doubt and it is 
possible to select between two courses, one of study and proposal and 
the other of build and test, the relative advantages of the two should 
be very carefully evaluated. If a prototype can be built for not much 
more than the cost of conducting an extended analysis, a prototype 
seems preferable because it can be expected to produce more reliable 
information thana study. If such information becomes available in 
advance of a production decision, the lead time between production 
decision and availability of operational combat systems will be lessened 
by the amount of "learning time" that would otherwise follow a decision 
to produce a combat system only defined on paper. Indeed, the per- 
formance of the prototype in land-based trials may not only enhance 
the validity of acquisition decisions, it may also solidify them, asa 


decision based onan existing prototype will generally contain less 





residual uncertainty than one based solely on analysis of engineering 
expectations. 

In essence, then, the construction of prototype land-based combat 
systems can provide a hedge against requirements (strategic) uncer- 
tainty. It does so by permitting a more certain resolution of techno- 
logical uncertainties than can design studies, no matter how elegant 
they are. The central fact is that under almost any conceivable circum- 
stances the transition from design proposal to first lot of production 
articles brings on change, and change introduces new uncertainties. 
Some are uncertainties that can be resolved with no particular difficulty 
while production continues. The function of a prototype is to permit 
the early identification of previously unrecognized problems and the 
resolution of recognized uncertainties that might, if they went undetec- 
ted, precipitate major changes in the performance, cost, or avail-. 
ability of specific weapon systems. 

Adhering to the principles described above would be difficult in 
anyecircumstamce. It isevery difficulttteday. The pressures for eanly 
commitment to production are enormous. The customer, in this in- 
stance, the Navy, is conditioned by the military environment to be 
less interested in the comprehensive resolution of uncertainty than 
in the early delivery of operational ships and combat systems. There 
is a natural tendency to assume that all problems will be little ones. 


In this situation the producer has very little motivation for investing 
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time and money ina prototype. Profits come from production. Firms 
are not paid very well for military development. Contractors would 
rather build what are incorrectly called "production prototypes" than 
"development prototypes, '' because a "production prototype'' implies 
a strong commitment to production while a ''development prototype" 
represents a proposal of which the faults, if any, are all too evident. 
Here again, whena "production prototype" is built the producer will 
be optimistic about the possibility of solving all problems early and, . 
as has been observed, the customer iS anxious to believe such re- 
assurances. 

Some critics and defense-industry experts recently (January 1974) 
have stated that a prototype, as a preproduction step, tends to blur the 
line between development and full production, enabling the Services 
to move systems into production a little bit at a time. In another vein, 
others argue that a preproduction prototype gives congressional critics 
more time to attack. The presumably higher price tag is an argument 
against building the ship/combat system which in turn is dysfunctional 


to the service who is trying to abide by directives and test before buy 


concepts. 


m. DOD, THE NAVY AND PROTOTYPING 
The DOD's current philosophy is that the use of prototypes must 


be increased in all development efforts to reduce costs and resolve 
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uncertainties prior to major production commitments. DOD has 
broken prototypes into three categories as follows: 

1. '‘ Experimental -- to demonstrate technological feasibility. 

2. Developmental -- to gain sufficient confidence before 
comitting a system to full-scale development. 

3. Production -- to prove out the system, tools and methods 
prior to quantity production. 

The DOD normally recognizes three approaches to prototyping, 
which are in most cases applied to experimental and developmental 
prototypes. They areas follows: 

1. Sole/single source -- prototypes developed and fabricated 
by a single developer. 

2. Competitive -- prototypes involving the same technology, 
developed and fabricated by two or more developers. 

3. Complementary -- prototypes involving different tech- 
nologies, developed and fabricated by two or more developers in 
response to the same basic requirement. 

The breakdown and extent of prototyping per se is broken out as 
follows: 

1. Component -- part of an assembly of parts to performa 
famction:. 


2. Subsystem -- a major functional assembly of a system. 
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3. Hybrid -- a system of equipment based upon an alteration 
or change to an existing system or equipment. 

4. ''Bare-bones'' model -- a system or equipment containing 
only its minimum essential elements. 

5. Complete system -- a combination of subsystems that 
makes up the ultimate system or equipment. 

As should be seen from the above, prototyping can take many forms 
and seek many ends. To achieve a completely successful and highly 
efficient set of prototype terminology is one of the important goals 
which the Navy feels must be achieved, if it is to be successful in im- 
plementing recent prototyping policies. The figure below depicts what 
appears to be the direction the Navy is going in its efforts to comply 
with guidance from DOD on prototyping and defense system acquisition 
in achieving a set (or body) or prototype terminology which can be _ 


utilized through the spectrum of Navy acquisition. 


TERMINOLOGIES 

Prototype Weapon Ships-conventional Ships-new 
Terminology Subsystems hull design hull design 
Experimental Bread board not Research 
Prototypes Models applicable Vehicles 
Development Advanced -land-based Test craft 
Prototypes development testing 

models / -test bed ships 

prototypes 
Production Pilot production Lead ship Lead ship 
Prototypes models/proto- 

types 
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A word of caution is in order when applying any so-called guide 
such as that in the previous figure. The purpose of sucha guide is 
to enhance understanding and communication, not to impose a rigid set 
of rules which will do more harm than good. The best example avail- 
able is probably that of shipbuilding, whereby a period of several years 
will be required for transition from present methods of shipbuilding to 


the sequential '"'fly-before-buy'' approach, even for ships in the 5,000-ton 
category. For larger ships such as carriers, the Navy must and will 
look for innovative ways to comply with the spirit of new prototype, test, 
evaluation and acquisition policies, but an idealized sequential program 
plan will not be attainable in all cases. 

Likewise, ‘the figure should not be interpreted to mean that each 
and every step must be accomplished in each and every program. As 
a general-rule, when determining whether certain steps may be omitted _ 
or not, development prototypes should not be required when the tech- 
nology has been "hardware'' demonstrated in some other program or by 
experimental prototypes. Neither should production prototypes be 
required when engineering change proposals of a minor nature can be 


made to "off the shelf" or "in production" defense systems of another 


Service. 


F. SUMMARY 
The central thesis of this chapter is that in certain circumstances 


it is sensible to build and test a prototype of a combat system before 





finally deciding to produce it in quantity. Such circumstances can occur 
if the technological risks inherent in the design are substantial and can- 
not be reduced to manageable proportions by analysis alone. The dif- 
ficulty of deciding when to adopt a prototype strategy arises from the 
necessity of deciding a priori how substantial are the technological, 
mission assignment, or source-Sselection risks. 

The circumstances under which the construction of a prototype 
combat system becomes a desirable element of the total development 
program include: 

| eee = program sufficiently well defined to permit developers 
to undertake the resolution of specific technological uncertainties that 
cannot realistically be expected to succumb to alternative techniques 
of uncertainty resolution (such as analysis and simulations) or that 
can be alternatively resolved only at greater cost than by the prototype _ 
route. (Build a complete ship only to discover that the combat system 
Apes a operate properly. ) 

2. When developers understand that the real purpose ofa 
prototype combat system is to identify necessary design changes 
quickly and accurately, and that one outcome of a prototype program 
can be a decision not to proceed to production. The object of a proto- 
type is to permit a prompt and economical decision in such matters. 

The real payoff which the Navy expects to achieve in prototyping 
programs can only be realized if thorough cost analysis and test and 


evaluation is conducted, reported and utilized by source selection 
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boards, project managers, contract negotiators, and high-level 
decision makers at every sequential step in the defense system ac- 
quisition process. The remainder of this thesis explores the efforts 


of the Patrol Frigate program to accomplish these objectives. 





V. THE PATROL FRIGATE'S TEST AND EVALUATION PROGRAM 


A. GENERAL 

The Patrol Frigate Ship Acquisition Program was chosen as an 
example of ongoing T&E and prototyping efforts because of the avail- 
ability of information, the cooperation extended by Project Office 
personnel, and several unique features of the acquisition program 
itself which lend application to subsequent ship procurement programs. 
The PF is also the first major ship acquisition program to come under 
the auspices of DODI 5000.1, DODI 5000.3, and additional T&E guide- 


lines developed in Chapter 3. 


B. PATROL FRIGATE SHIP HIGHLIGHTS 
kaw Objectivesof thesPEyProgram 

The objective of the PF program is to acquire for the late 
1970's-1980's timescale a class of ships capable of ensuring our use 
of essential sea lines of communications. It is intended that these 
ships will provide, at least cost, the necessary improvement in the 
Navy's surface combatant capability to: 

. Defend non-carrier forces against airborne forces 

- Conduct ASW operations 

These operations will be conducted in conjunction with other 


sea control forces. 
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2. Ship and Program Constraints 

The PF ‘i an austere escort ship designed toward the major 
program goals initially imposed by CNO, as follows: 

. Average follow ship cost (without escalation) not exceeding 
$45.7 million (FY 73 dollars). 

~ Full load displacement not exceeding 3400 tons 

2 Total personnel accommodations not exceeding 185. 

In order to meet the cost constraints, weapons and sensors 
requiring research and development have not been included in the PF 
design, at least intheory. Ensuring approval for service use of major 
individual equipments and systems has been made the responsibility of 
the Systems Commanders, NAVSHIPS, NAVORD, etc. 


It is intended that the PF will operate in conjunction with other 


sea control forces and not as an independent unit. Hence the, PE has,.,..., . 


been designed to supplement the capability of other ships rather than 
operate ina stand-alone capacity. 
3. Mission Requirements 

To ensure our use of essential sea lines of communication 
against the contemporary and postulated threats along with planned 
and existing escort forces, Development Concept Paper #97 of 3 April 
1973 requires the PF to be able to: 

= Provide self-defense and operate as part of an ASW 
screen defending the escorted force or operate on picket station in 


support of the escorted forces. 
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= Operate in conjunction with other escorts to provide area 
and close-in AAW defense of the escorted force and self-defense 
against aircraft and airborne anti-shipping missiles. 

g Provide for defense of the escorted force and own ship 
against hostile surface ships to over-the-horizon ranges. 

e Operate in conjunction with other forces in open ocean 
ASW operations and subsurface surveillance. 

= Supplement area AAW forces in the detection and engage- 
ment of airborne threats. 

“: Einforce blockades and operate offensively in conjunction 
with other forces against mercantile shipping and naval forces. 

= Provide for SAR, surveillance and evacuation operations 
and establish a naval presence. 
EY. The PF is not intended to serve as a carrier force escort. 
or in high-threat areas. 

4. PF Combat System Components 

Systems selected for the PF consist of the following: 
WEAPON 
-Standard Missile (Surface-to-air) 
-Harpoon Missile (Surface-to-surface) 
-~76mm Oto Melara Gun 
-Space and weight for Close In Weapons System (CIWS) 


-~Mk 46 Torpedo 
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LAUNCHER 

-~Mk 13 Guided Missile Launcher (Fires Harpoon or Standard Missiles) 
- Mk 32 Torpedo Tubes 

FIRE CONTROL SYSTEM 

-Mk 92 (STIR) Two Channel System 

SENSOR 

~SPS-49 Air Search Radar 

-SPS.55 Surface Search Radar 

-SQS-56 Sonar 

-Passive ESM System 

COMBINED SENSOR AND WEAPONS DELIVERY 
-Two LAMPS Helicopters 


The required capability of the above systems elements has 


--been-previously established and. it is required that their integration ... 


into a single combat system shall in no way adversely alter these 
established capabilities. 
5. PF Procurement Phasing 

A — has been awarded to Bath Iron Works Corporation 
to build the lead ship. Contracts for follow-ship procurements are 
tentatively to be in two batches of 24 and 25 each over a five-year 
period. Ship deliverics are to be at the rate of about one per month, 
with the lead ship being delivered in June 1977 and the last ship of the 


second block in the 1981-1982 timeframe. 





C. PATROL FRIGATE T&E PROGRAM 

The T&E phases for the PF are considered to be Developmental 
Test and Evaluation (DT&E), Operational Test and Evaluation (OT&E) 
and Production Acceptance Test and Evaluation (PAT&E). OT&E is 
further broken down into Initial Operational Test and Evaluation (IOT&E) 
and Follow-on Test and Evaluation (FOT&E). 

DT&E is viewed as a class of tests and evaluations for the specific 
purpose of facilitating the evolution of the ship system and is conducted 
under the sponsorship of the Developing Agency. For the PF, DT&E 
is that portion of land-based test site testing concerned with installation 
and integration of the combat systems in addition to that developmental 
testing to be conducted in support of the individual equipments. Integra- 
tion of the combat system is considered to be a "Risk Watch List" item, 
«.1,.e., failure to achieve satisfactory integration at the test site.would. 
adversely affect DSARC III presentation. Integration of the Combat 
System is to be proved at the test site prior to DSARC III. 

IOT&E for the PF consists of two separate areas. First, the 
individual equipment/systems of the 'Risk Watch List"! in the DCP 
are to undergo separate IOT&E's. For instance, the Mk 92 fire con- 
trol system is scheduled to undergo IOT&E on the USS TALBOT DEG-4 
in September of 1974. Second, the integrated systems will undergo 
IOT&E at the land-based test site. 'Risk Watch List'' systems which 


will undergo individual IOT&E are: 
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-Mk 92 FCS/MK 75 gun mount and 76 mm ammunition 
-AN/SPS-49 Radar 

~-AN/SQS-56 Sonar 

~HARPOON Fire Control System 

The responsibility for initiating IOT&E for these systems lies 
with the participating managers (PARM!'S) managing the development 
and procurement of their respective equipment. However, the T&E 
Manager of the SHAPM's staff will closely monitor the test programs 
to insure that PF test schedules can be met and that any problems or 
deficiencies during the IOT&E's are taken into account. 

IOT&E of the integrated systems at the Combat System Land- 
Based Test Site (CSLBTS) is scheduled to begin in January 1975. 
Naval personnel will be used to evaluate the systems. Accomplish- 
ment of adequate IOT&E at the LBTS with satisfactory results.isa,. 
prerequisite for approval by DSARC of follow-ship procurement. As 
such, the events and planning leading to IOT&E as wellas IOT&E 
itself will receive the closest.attention of the OT&E Organization. 

FOT&E will be addressed in detail by an FOT&E Plan which will 
include the possible requirements and procedures for assigning the 
lead ship to OPTEVFOR for a complete at-sea operational appraisal 
prior to the ship's release for Fleet usage. This requirement was 
embodied in the DSARC I/II decision of 27 September 1972 which 


stated in part: 
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The Navy is requested to develop a plan for, and evaluate the 
impact of assigning the lead PF to OPTEVFOR for a reasonable period 
to complete an at-sea operational appraisal of the PF as a whole prior 
to the lead ship's full release for Fleet usage. This plan and evalua- 
tion, together with the Navy's recommendations, should be submitted 
to OSD at the time of preparation of the revision to the DCP for initia- 

_. tion of construction of the first follow ship. 

The impact of this assignment will be evaluated and an evaluation 
report will be prepared by the T&E Planning Group for submission to 
the CNO. The FOT&E Plan and the evaluation report will be com- 
pleted prior to the DSARC III milestone. 

PAT&E for the PF is functionally separated into three areas. 
These areas are documentation certification at the LBTS's, acceptance, 
testing and documentation validation on the lead ship, and acceptance 
testing onthe follow ships. Detailed information on the PAT&E is 
contained in the Ship Test Management rian of 15 November 1972 

Documentation certification at the LBTS's will consist of actual ....._ 
conduct of test procedures for the installed equipments/systems. 
Such use will enable test procedures to be corrected if needed and 
certified prior to use on the lead ship. More will be said about 
"certification" ina later section of this thesis. 

On the lead ship, the certified procedures will be used to conduct 
the tests leading to validation of the procedures and acceptance of 
the shipbuilder's efforts. Test procedures for those equipments and 


systems which were not certified at the LBTS will also be validated 


on the lead ship. The validated documentation from the lead ship 
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will then be supplied to follow shipbuilders. Follow-ship testing will 
use validated test procedures for test conduct leading to Navy accept- 


ance of the shipbuilder effort. 


D. PATROL FRIGATE TEST ORGANIZATION 
The Patrol Frigate Ship Acquisition Project Manager has established 
a Total Ship Test Program, and the organization for its accomplishment 
for the Patrol Frigate. The organization that will be used by the Patrol 
Frigate to develop test documentation is shown in Figures land 2d. The 
principal members of the organization are identified in this section. 
Specific responsibilities are elaborated in the next section. 
1. Ship Acquisition Project Manager 
The Ship Acquisition Project Manager (PMS-399) is respon- 
sible to the Commander, Naval Ship Systems Command and the Major 
Surface Combatant Ship Project Manager (PM-18) for the Patrol Frigate 
Ship Acquisition. 
2. Test and Evaluation Manager 
The Test and Evaluation Manager (PMS-399T) is responsible 
to the SHAPM for the successful conduct of the PF Total Ship Test 
Program. 
3.) Beteteohnip Test Director 
The Supervisor of Shipbuilding at Bath, Maine has been 


designated as the Total Ship Test Director and is responsible for the 


76 





uoTJezjuesIO yay diyg ayestrq jomeg “tT osan3y7 


SY3LLVW 1S31 dJIHS NINOILD3YNIO ——@8-——-@e— 
NOILYVNIOYOO) —— — — 





ON39371 








ALOV TIHd D3SAVN 
¥O193410 
IN3Wd0O13A390 1S3l 
W31LSAS SNIYS3SNISN] 


4391340 1L93f°0ud (MIG) 






31IS 1331 O3SVE-ONVT 
yIO TING! 
W31LSAS NOIS1NdOud ae ae 





ES} 3009 HLVSE dIHSdNS 
yOLI3YI0 
1S$31 diIHS 1V1L01 


ZL66E SWd en ee ee 166 SW 

YAWN 
NOILWHO31NI path 
a a NOILYN TVA] 8 1S3L 


W3LSAS NOISINdOWd 






66E SWd 


BIOVNVW LO35fOud 
NOItLISINOSY diHS 


SINI9V 
IN3Wd013A30 1S31 


SYIOVNVW 
IN3WdO19A30 1S31 


OS6? HIOSYON S3ASMSN 
YOLI3YIO 
1IN3Wd013A30 

1831 W31SAS LVSWOD 





(WSS) 

YOLIVYELNOOD 
LYOddNS NOILVYSDILNI 
W31SAS LY8WOD 


aLIS 1$31 Q3ISVa-ANV1 
WALSAS LVEWOD 
JOYVHO NI Y391550 





£L66E£ SWd 
YIOVNYW 


NOILVYSIILNI 
W31SAS LVSWOD 





fare 





Cee tis lAvi 
Q1i0 22E IAI womens 
erewnD oy IKE GE pwd} 
fOr tee evepesg ON 
Gs SeteraAWN cealae 


iret Coe lAvi 
Gpnceeg Ww QUOT 
19C28 PTD) AIO VI DAtAWN 
OG4 wer rg Oe sertu 3 





MTOOGS IAY! 

veason 2 3 

(9099 sPoD: 
AIQHONIJISA YN 
Sco, tay HUONIIMD 


4S09 OF 1A¥I 
wiwarg F 


(OPE e OOD) STSMSN 


SNO1L¥ DINO rewOD 


are CevlAvi 
tare g 


whe JIM RTA 


NOUV DIAN 


orse CSZ IA! 


mut yedinies te 
AIO 453m 
ve 75459 333ANN FIDO SB US EEO L 
N¥IWL S135" 


ELE SR (Aw) 06% Cee LAV! 
Sea wens g 
$620 OD) 439453N ene osen 
aa ipy bumues jerUA 


OElR 222 IAvi Wecvw20 £22 sAvi 
usw tv ate Vi 
1941601 M2 13AWN IGZeLCe ePODl BIVATA 
WOL N27 12AVN WO UIVATAN 


(Tete Wess YIOw +04 I O44 GLP IAW) 
een f wODt 
(TE. OPO) HIVE Bo lHteNe 
O18 HO Mo, Or ys tle, 


W/COOOZ7Z AYE 
218v2Z fr wood 
Aeet tres 
(mtn a Bemthadtd teat Lanett? § 





Syuasy/Siaseurw yuaudojaaag 4Say a3eSliq [oyed °Z s2insry 


$600 000 1Avi 
‘5 MM 


(SOE HE Pe), 
AO OM DISA WN 
WW WTNY 








4222 Wa 409) 
pone amy, tf 
On7) vem Ary 
@ Win 


eGri S62 1A¥i 
tum ¥ 
31919 DISAYN 
Se 2aIwNieds ANIZVOVA 
BOWDEN DIS TYNSIA 


@ ict 942 1AW! 
wt eaves) ¢ 
OS618 DISAWN 
1g -ers 


S994 068 AWE 
ens oO 
1ECSE eeoD) 
te g IN 
108 2 ov MSY 


rOci 942 1AV! 
sy 01S Ad 
1098219 DISATN 
wIsINOMING VY OwAD 





$206 16 31 OOrl PLZ IAW 
Oe 1 
ZOOL pth eGo} 
* 87 OVN ten 
Weve veoam 


Otg1 SOL AY! 
eevrelT ye 2 
1030¢19 DISAYN 
I¥eO ind 


Codt $62 1A éefi OFF 1A¥I 
wo wt Sad ane | 
1088619 DISAYN £299 €POD) 5 7SaaSH 
Swuv iv SiwO Ca wee 


Oris SL AYE 
Obs eiora, fF 
VO3GC 19 DISAWN 
ee ae ee oe | 
sume Hun) peu @iug 


CeO Gez IAW! 
ieee 
139 ope)! ve Mush yaar 
vA SL OA 





6091 962 [AWE LABOLs Lh1O OLS 19068 

v@.g0 “11 wet 5 

OZi19 DISAVN S55 4mry 
voted FH rwwe) Sdale ure 


vers 1S AVI 
wees ot 
if (See me > 
4 1J00ON 5S ISMSS 
WO] QHOAYN 


4/0001 9G2 1A¥) 
iuews) oy 
C0 46248 DISAVN 
WO4 DISAYN 


SvVAS (PB LAY! 
wren 00 
1b bS@e POD 4 10 OM SDE 
OO werss swe 








78 





development of ship test documentation and its validation on the Lead 
Ship. 
4. Engineering System Test Development Director 

The Naval Ship Engineering Center, Philadelphia Division has 
been designated as the Engineering System Test Development Director, 
with the responsibility for the development of the Hull, Machinery and 
Electrical (H/M/E) test documentation. 

. 5. Combat System Test Development Director 

Naval Ships Weapon System Engineering Station, Norfolk 
Detachment has been designated as the Combat System Test Develop- 
ment Director with overall responsibility for the development of Com- 
bat System test documentation. 


6. Test Development Managers 


~<"The Test Development Managers under direction of the Combat. . 


System Test Development Director, provide test documentation for 
specific subsystems of the Combat System. 
7. Combat System Integration Manager 
The Combat System Integration Manager, PMS-399.13, is 
responsible for successful integration of the elements of the PFCS 
and the provision of system level test documentation for the testing 
of the lead and follow ships. 
8. Officer-In-Charge Combat System Land-Based Test Site 
The Officer-In-Charge of the Combat System Land Based 


Test Site coordinates activities at the Combat System Land Based 
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Test Site with specific responsibilities as described in the next 
section. 
9. Lead Shipbuilder 

Bath Iron Works (BIW) has been selected as the Lead Ship- 
builder. In that capacity the preparation of various items of test 
documentation and ship test performance are within their responsibility. 

10. Combat System Integration Support Contractor 

The Naval Ship System Command has entered into a contract 
with Sperry Systems Management, Sperry Rand Corporation, as the 
Combat System Integration Support Contractor to assist the ee 
System Integration Manager (CSIM) in the providing of system level 


test documentation. 


E. PATROL FRIGATE TEST ORGANIZATION 
Pee at voduction 

The PF SHAPM has established a Total Ship Test Program 
for the Patrol Frigate in accordance with existing regulations and 
which appears in the Patrol Frigate Ship Test Management Plan. This 
program is to provide for comprehensive tests which will determine 
the status of equipment readiness during construction and the life-cycle 
of the ship. The organization that will be used by the PF to develop 
test documentation is identified in the previous section. Detailed 
responsibilities of the actors involved in T&E of the PF as presently 


viewed follows. 





2. Ship Acquisition Project Manager 
The SHAPM provides guidance as necessary to the T&E 
Manager in the areas of planning, budgeting, developing, and apprais- 
ing ship test documentation. The SHAPM is responsible for the PF 
Ship Acquisition Program and will provide guidance as necessary to 
the T&E Manager for the implementation of the Total Ship Test Program 
rot bP). 
3. Test and Evaluation Manager 
The Test and Evaluation Manager (PMS-399T) is responsible 
for the successful conduct of the Patrol Frigate TSTP with specific 
responsibilities for: 
-Successful conduct of the T&E Program 
-Generation of the test sections of the Follow Ship 
Specifications 
-Test of the Lead and Follow Ships 
-Development and maintenance of the Test and Evaluation 
Master Plan and Ship Test Management Plan 
-Analysis and feedback of the results of testing into ship 
design 
-Development and maintenance of documentation for the 
testing of Lead and Follow Ships 
-Compliance with T&E policy established by the Major Sur- 
face Combatant Ship Project Manager (PM-18) and the 


Commander, Naval Ship Systems Command. 
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4. Total Ship Test Director (TSTD) 

The Total Ship Test Director is responsible to the SHAPM 
through the T&E Manager for coordinating the development ship test 
documentation and its validation on the Lead Ship. 

The TSTD has specific responsibilities to: 

~Manage the development of an Integrated Test Package to 

ensure that the Follow Shipbuilders comply with the Ship- 
building and Performance Specifications. 

~Manage the development of a test package for the life-cycle 

‘ testing of the PF. 

-Prepare and forward Quarterly Progress Reports and 

Exception Reports. 

~Supervise the conduct of testing of the Lead Ship during 

construction at Bath Iron Works. ; — 

~ Validate Test Procedures during the testing of the Lead Ship. 

ikees the review and revision of the test sections of the 

Lead Shipbuilding Specifications and the Test Documentation 
Booklet for use in the Follow Ship Request for Proposal. 
~Direct the activities of the Combat System and Engineering 
System Test Development Directors in accomplishing the 
above tasks. 
~Submit schedule and budget data to the PF T&E Manager. 
.Maintain a configuration control program to ensure avail- 


ability of correct test procedures for Lead and Follow Ship 


testing. 
82 





Engineering System Test Development Director (ESTDD) 


The ESTDD has specific responsibilities to: 


-~Accomplish tasks assigned by the TSTD for development of 
the Test Index and Test Outlines for Contractor Furnished 


Equipment for the Lead and Follow Ship Test Programs. 


~ Provide technical assistance for liaison and monitoring 


during Engineering System test documentation validation on 
the Lead Ship. 
-~Many other duties relating to H/M/E tests beyond the scope 


of this thesis. 


Combat System Test Development Director (CSTDD) 


The CSTDD has specific responsibilities to: 

-~Accomplish tasks assigned by the TSTD for development of 
Combat System test documentation for the Lead and Follow 
Ship Test Programs. 

-Provide for on-site liaison and monitoring during Combat 
System test documentation validation on the Lead Ship. 

-~Review the Integrated Test Package and recommend which 
Combat System Test Procedures are suitable for conversion 
into Maintenance Requirement Cards for inclusion in the 
Planned Maintenance Subsystem. 

-~Provide for on-site witness of the conduct of tests at the 


CSLBTS and certifying the correctness of Test Procedures. 
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-As directed by the TSTD, participation in the review, and 
recommend revision, of the test sections of the Lead Ship- 
building Specifications and the Test Documentation Booklet 
applicable to the Combat System for use in the Follow Ship 
Request for Proposal. 

-Monitor development of Combat Systems Test Procedures and 
outlines. 

aRewew all Combat System test documentation to ensure its 
technical accuracy and adequacy to support the ship test 
process. 

-~Prepare schedule and budget recommendations for the Lead 
Ship Test Program. 

7. Test Development Managers (TDMs) 

The TDMs are responsible for organizing, planning, managing, 
and controlling the development of specific subsystem test documentation 
for the Combat System. The TDMs are responsible for: 

: Managing the development of all equipment and subsystem 
test procedures for equipment under their cognizance. 

Rey ievany Test Procedures for technical accuracy and 
content. 

-Managing the development of special and at-sea trial Test 
Procedures for external communications, ASW systems, 


LAMPS, and Harpoon Weapon System, where applicable. 
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-~Ensuring the generation, review, and modification of the 
Test Index, .Test Outlines, and Test Sequence Networks for 
equipment under their cognizance. 

Wines directed by the CSTDD, acting as certifying agents at 
the CSLBTS and validating agents on the Lead Ship for the 
certification and validation of test documentation for equip- 
ments under their cognizance. 

-Providing solutions to test problems encountered during the _ 
conduct of testing at the CSLBTS and onboard the Lead Ship. 
~The Officer-In-Charge (OIC) CSLBTS and the TSTD have 
been given authority by the SHAPM to make interim changes 
to Test Procedures during testing at the CSLBTS and onboard 
the Lead Ship respectively. 

8. Combat System Integration Manager (CSIM) Laas, _ 

The CSIM's principal function is to insure the successful 

eer of the PF Combat System. His primary responsibilities in 
support of the Ship Test Program, areas follows: 

-Provision of special and at-sea trial Test Procedures for 
certification at the CSLBTS and for eventual Lead and Follow 
Ship testing as applicable. 

-Revision of CSLBTS Test Procedures for Lead Ship applicabil- 
ity and generation of additional Test Procedures for Lead 


Ship as required. 
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-Development of Test Procedures for Combat System com- 
ponents installed in Follow Ships and not in the Lead Ship. 
9. Lead Shipbuilder 
The Lead Shipbuilder is responsible for the following Ship 
Test actions in accordance with his Contract: 

-Review the Test Documentation Booklet and prepare revisions 
to equipment, subsystem, system and special test outlines as 
necessary to ensure accuracy and adequacy of Ship Test 
Program. 

-Develop Test Pfocedures for Contractor Furnished Standard- 
ized equipments. 

-Develop Test Memoranda for CFE. 

-Conduct Combat System equipment, subsystem, system and 
special test procedures onboard the Lead Ship. Deficiencies 
shall be brought to the attention of the SUPSHIP, Bath. 


10. Officer-In-Charge Combat System Land-Based Test Site 
(OIC CSLBTS) 


The OIC CSLBTS is responsible to the SHAPM, through the 
Technical Director, for the overall management and operation of the 
Navy assets at the CSLBTS. In support of the Ship Test Program, he 
has the responsibility to: 

-Demonstrate, where feasible, Combat System test procedures 

at the CSLBTS for certification prior to use on the Lead and 


Follow ships. 
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-Schedule tests and demonstrations. 
-Supervise the evaluation of proposed changes to Combat 
System Test Procedures at the LBTS. 
ll. Combat System Integration Support Contractor 
Test responsibilities of Sperry Systems Management as the 
Combat System Integration Support Contractor includes: 
-Develop unique subsystem, system and special Test Pro- 
cedures for the Combat System, initially for use at the 
LBTS and eventually for use during the Ship Test Program. 
aeeane maintenance assistance and engineering support 
during certification of equipment, subsystem, system and 
special Combat System Ship Test Procedures at the LBTS. 
-Evaluate proposed changes to Combat System Test Pro- 
cedures at the LBTS and forward results with recommenda-_ 
tions to the TSTD via the CSTDD. 
-Provide the TSTD technical support if required during the 
conduct of Test Procedures on the Combat System for both 
the Lead Ship and Follow Ships. 


Highlights of the present Navy contract with Sperry Systems 


Management will be provided in a subsequent chapter of this thesis. 


F, CONCLUSIONS 
Requirements set forth in DODD 5000. 1 and current Navy Depart- 


ment Directives provide the required Patrol Frigate test and evaluation 
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guidance with which to establish a ship/system T&E approach which 
demonstrates, at various decision points, sufficient confidence for 
follow-ship commitments. The PMS 399 organization for T&E has the 
capability of providing for successful acquisition of 50 mission- capable 
Patrol Frigates with performance-validated ship's crews, ship's 
hardware and software. 

Effective implementation of the PF T&E program concept requires 
centralized authority and responsibility in one agency acting for and 
under the direction of the SHAPM. This has been provided by a 
Test and Evaluation Manager. One caveat however, is in order and 
is in reference to behavioral considerations of participative manage- 
ment as employed by the PF T&E organization. The SHAPM's T&E 
organization is predominately a military/civil service organization 
and its managers have had a lifetime in military and government affairs... . 
In such organizations it is oftena rule not to bring up problems that do 
not already have apparent solutions. The essence of participative 
management in the PF T&E program is openness and freedom of 
exchange of both problems and problem solutions. Centralized top 
management seeks this, but it does not come naturally and requires 
perceptive treatment by all who are involved. 

Another area in which the PF program is vulnerable stems from 
the ever increasing number of Congressional, DOD and Navy-mandated 


programs. Previous chapters have dealt with T&E aspects. Similar 
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forces are at work in other areas such as integrated logistic support, 
value engineering, surplus labor area selection, small business with- 
holds and shipbuilding. Across the board PMS 399 is particularly 
vulnerable to those programs which are aimed at increasing centraliza- 
tion of weapon system procurement. While PMS 399 will support these 
programs, they have a significant impact on planning because they are 
often of great scope, unfunded, absorb scarce and crucial resources, 


and decrease the SHAPM's planning self-determination. 





VI. THE PATROL FRIGATE COMBAT SYSTEM 
LAND BASED That SULE 
A. GENERAL 

Land-based integration of the Patrol Frigate combat system is a 
keystone of the PF T&E program. Prior to the production decision 
(DSARC III), presently scheduled for March of 1975, the Land-Based 
Test Site will be used to prove installation procedures to the maximum 
extent practicable and to prove the mechanical and electrical compat- 
ibility of the major equipments of the combat system. Additionally 
the Combat System Preliminary System Operational Program will be 
demonstrated at the CSLBTS. The Final Operational Program will be 
debugged at the LBTS, after DSARC III, and will be demonstrated on- 
board the lead ship after delivery. 

Proofing of computer programs will be emphasized as a major 
factor in achieving full utilization of the integrated combat system. 
Engineering and functional analysis at the LBTS will lead to early 
solution of integration problems that may be exposed. 

An attempt will be made to address critical test and evaluation 
questions and issues identified in the DCP such as: 

-Are the individual combat subsystems which are to be incor- 
porated in the ship technically acceptable and operationally suitable 


and effective? 
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-Is the integration of the various combat systems through their 
respective interfaces adequate, in terms of data transfer characteristics 
such as handling capacity, rate and quality, to meet operational 
requirements? 

Eas those operational characteristics of the integrated combat 
system which can be estimated based on initial operational test and 
evaluation, including reliability and maintainability test and evaluation, 
sow a reasonable probability that the ship class, when afloat, will be — 
able to satisfactorily accomplish the mission for which it was designed? 

The use of a land-based test site as an aid in ship procurement is. 
not a new concept. As early as 1958 an ashore navigation center was 
constructed by Sperry Systems Management Division in support of the 
SSBN 598 class of submarine comegancne Test sites employing 
specific equipments are presently available at additional locations _ 
such as Mare Island. However, the PF CSLBTS is the first site con- 
structed which is dedicated to a single class of ship. 

Recent experience with the DLGN-36 class procurement has re- 
inforced the need for a dedicated test site. A review group under the 
banner of the CNM recently (14 March 1974) completed an audit of the 
problems found in the DLGN-36 combat system. Throughout the report 
it is repeatedly pointed out that the lack of a LBTS seriously hampered 


and delayed successful combat system integration. 





For the DLGN-36 procurement and others (DD-963, DE-1052 etc. ), 
land-based testing was conducted ona fragmented subsystem basis. 
Some subsystems were partially tested at Mare Island for instance, 
and others at Dam Neck. In neither case was there a "full up" capa- 
bility for testing the complete combat system. The first time the 
major subsystems were all brought together at one place was aboard 
ship. Since there was no land-based test site, the full up system 
problems were not encountered until the advanced stages of shipboard . 


/ 


testing. Such is not the case for the PF procurement program. 


B. PATROL FRIGATE LAND-BASED TEST SITE PHYSICAL PreATHE 
The LBTS is located at MacArthur Airport, Islip, New York. Its 
primary initial function is to serve as a facility for the integration of 
the combat system equipment and for training a U. S. Navy crew to 
demonstrate the system prior to the PF production decision. It is 
anticipated that, following the production decision, the LBTS will be 
used for crew training, shipyard personnel training, computer program 
verification, change verification, test program checkout, etc. The 
LBTS will also be used for design and test of system modifications. 
The building which houses the LBTS contains the physical mock-up 
(PMU) compartments for the PFCS, the test facility for the Mk 92 FCS, 


and facilities for support of these activities. In addition, the LBTS 


contains the offices and classrooms required to support this plan. 
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Appendix B depicts the location of the test site in relation to MacArthur 
Field, a layout of the test site, a plan view of the CIC and sonar control 
room, the CIC, IFF and sonar equipment room, the Mk-92 equipment 
room, the STIR (Separate track and illuminate radar) equipment room, 
the operations center (OCC), the program checkout area (PCA), and 
the Mk-92 test site. Radar antennas are physically located on the 
northern side of the test center in the approximate configuration that 
they will appear onboard ship with the exception of height which is a 
limited by FAA regulations. 

~" °“-The physical mock-up compartments will contain actual operating ---->: 
equipment as well as simulators and mock-ups, and will represent the 


precise configuration of the ships compartments. The CIC physical 


mock-up contains functional communications: circuits to enable the 


’~ simulation of tactical operations. The LBTS also employs functional,, .. 


SPS-49, SPS-55 and Mk 92 Mod 2 CAS (combined antenna system) and 
STIR radars. It is thus capable of detecting and tracking actual 
aircraft. No capability exists, however, for actual subsurface targets 
and only a limited capability exists for surface targets. Ships motion, 
as wellas air, surface, and subsurface targets can be simulated by an 
OCC Console and inserted into the command and control element of 
the combat system. Based on these simulated data inputs, the combat 
system will develop CIC console scope presentations and data readouts 


depicting test or tactical operation scenarios. Operator actions will 
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result in selection and ordering of simulated weapons. During the 
performance of the exercises, CIC operator actions and displayed 
console data can be monitored, recorded, and evaluated through use 
of closed circuit TV, PPI displays, data readouts, and recording 
equipment on the OCC console. 

The PMU is realistic to the point of being deceptive, and only the 
lack of ships motion and the smell of salt air detract from almost 


complete authenticity. 


C. USES ENVISIONED FOR THE CSLBTS 

1, DSARC III Demonstration 

As previously stated, the primary short-range use of the 

LBTS is to prepare a presentation for the production decision at 
DSARC III. In fact, there are those close to the PF project who 
believe that the production decision hinges almost completely on how 
well the CSLBTS reacts to the scenarios during the presentation. In 
preparation for this demonstration, a Navy crew is presently under- 
going training under Sperry guidance to gain the knowledge and skills 
necessary to demonstrate satisfactory operation of the PFCS during 
the IOT&E demonstration. A limited number of COMOPTEVFOR 
personnel will also attend this training. 

Zee Ee nine 


This training includes the following: 
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-The characteristics of the PFCS and the LBTS, to include 
system capabilities and limitations, mission, combat system 
elements, subsystem elements and manning. 

-Combat system functional data flow, including the signal 
characteristics and time relationships. 

~The IOT&E tests to be performed on the PFCS including 
threats, test method and test procedures. 

. The theory needed to support and understand the performance 
of operational procedures, for detection, identification, 
evaluation, weapon assignment and weapon control under 
various threat conditions. 

: The theory needed to perform operational trouble analysis 
and fault isolation to the subsystem/equipment level by 
employment of system operational tests and programmed. 
operational functional appraisals and by logical analysis 
sears maintenance documentation. 

-Documentation and procedures to support system level 
organizational maintenance. 

-Ability to perform normal and casualty operations onan 
assigned station with an acceptable probability of successful 
target engagement, for each of the following scenarios: 

s-rower up, Condition 3. 


--Work station checkout, Conditions 3 and l. 
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~-Single aircraft attack, Conditions 3 and l. 
-~-Single surface/near surface attack, Conditions 3 and l. 
--Single subsurface attack, Conditions 3 and l. 
--Multiple threat, Conditions 3 and l. 
-~-Casualty mode, Conditions 3 and l. 
-Ability to perform trouble analysis and fault isolation of 
PFCS faults to the equipment level, using diagnostic computer 
software and maintenance manuals. 

-Ability to perform system level organizational maintenance. 

' -~Ability to perform the IOT&E, with acceptable accuracy 
and within specified time limits, using specified methods 
and procedures. 

The objectives of the lead shipbuilder course is to familiarize 
shipbuilder personnel with the PFCS and to provide them with the unique 
skills and knowledge required to install, integrate, and check out the 
combat system onthe Patrol Frigate. The follow shipbuilder course 
will be the same as the lead shipbuilder course except for modifications 
to accomplish curriculum improvement, as necessary to reflect ship- 
board configurations, or as required by the follow shipbuilders. 

As a training vehicle, the LBTS is limited by the following 
considerations: 

-Until U. S. Navy certification 15 December 1974, priority of 

use of facilities and equipment is assigned to engineering 


development, system integration, and testing. 
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-After 15 December 1974 and until IOT&E is completed, 
priority of use will be assigned by the U. S. Navy OIC of the 

* EBS. 

-The LBTS equipment complement does not include all ship- 
board CIC equipment. Simulation is provided in lieu of major 
system inputs such as sonar, ECM, ships motion, and weapons 
dynamics. Mock-ups are provided for major equipment found 
in the sis configuration but not essential to combat 
system IOT&E. 

- Unscheduled (corrective) maintenance will be performed by. .- 
Sperry engineering and contractor technical support rather 
than U. S. Navy personnel. 

-Deliberately faulted modules will not be available or authorized 
for use in the system prior to or during the IOT&E, _, 

3. Generalareas 
The specifics of the majority of the tests to be conducted at 
the LBTS are classified. However, the LBTS will be used as a test 
bed to address the following type of questions: 

-Can the PF develop attack data on submarines by the use of 
active sonar while maintaining speeds up to X knots? 

-Can the PF decoy passive acoustic homing torpedoes? 

-Is the PF capable of employing ASW weapons to the effective 


range of installed subsurface sensors? 
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-Is the PF capable of controlling HS/VS/VP aircraft? 

4e the PF capable of Aevecting airborne threats by use of 
radar? 

-Can the PF detect and evaluate electromagnetic radiation? 

-Is the PF capable of tracking and engaging fighter~bomber 
type aircraft? 

-Can the PF track and engage air, surface or sub-surface 
launched anti-shipping missiles? 

- the PF capable of engaging surface targets to over-the- 
horizon ranges? 

-Is the crew capable of performing all combat systems 
operations? 

-Is the crew capable of performing all combat system 
maintenance tasks? 

Documentation 


One of the most difficult and expensive areas to manage in 


the procurement of a new combat system is documentation, especially 


at the system level. The PFCSLBTS will aid documentation as follows: 


-Generation of publications engineering data and text. 

- Verification of system documentation for builder and fleet 
use 

-System Documentation 


-~- Technical manuals (system and equipment) 
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--otandard operating procedures 
--SMP's (System maintenance procedures) 
--PMS (Planned maintenance system) 
--System computer program and documentation 
--sSystem and equipment diagnostic computer programs 
-Data flow and one-function diagrams. 
5. Logistic support 
Logistic support for equipments in combat systems are often 
based on engineering data and test bench operation by equipment manu- 
facturers. For the PF there is a Failure Data Collection, Analysis, 
and Corrective Action Plan which is designed for the unique require- 
ments of the Land Based Test Site while permitting the acceptance of 
data from vendor facilities, shipbuilders, government sources and 
eventual fleet operations. This plan incorporates failure data collec- 
tion, failure analysis, corrective action, and failure reporting. Thus 
it may be Sis as a dual-loop system whereby better logistic support 
requirements may be determined initially based on LBTS data and in 
the long run ona system whereby continuing assessment of support 
requirements based on fleet data may be made. 
6. Long term uses 
Sperry's experience with the SSBN 598 class ashore-navigation 
center seems to indicate that in the short run the configuration ofa 


given LBTS closely duplicates the class of ship for which it was built. 
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As time goes onand other classes of ships come into being, there isa 
tendency to modify existing LBTS to accommodate ashore testing of 
new systems. Inthis regard, the LBTS is a vehicle to groom tactical 
equipment, and verify installation test procedures and system computer 
programs. It supports system improvement studies, development of 
new system concepts, and analysis of advanced system computer pro- 
gram concepts. In development areas, the LBTS may be used to test 
and evaluate new operational techniques, system-equipment changes, 
define change requirements, evaluate proposed changes, and to verify 
shipalt/ordalt change documentation and accuracy. 
7. Land-based Test Site Users 

While the total spectrum of users of the PF CSLBTS can only 
be determined after the fact, it is envisioned that many types of activi- 
ties will use the LBTS to advantage. Users envisioned at the present 
time include procurement decision makers, system designers, system 
analysts, system programmers, equipment contractors and sub- 
contractors, publication agencies, integrated logistic support personnel, 
fleet personnel, field engineers, and shipbuilders. In addition, relia- 
bility managers, those involved in quality assurance, and politicians 


are sure to make use of the facilities available at the LBTS. 


Dy CextaOr Tht PATROL FRIGATE LAND-BASED TEST SITE 
The initial goal of the PFCSLBTS is to adequately demonstrate the 


viability of the proposed combat system to the DSARC at the production 
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decision and thus support fleet requirements in the 1980's and 90's. 
The other side of this goal, so to speak, has a negative connotation. 

To wit, if the system is not viable, it will not be procured. Within the 
initial goal, there are many subgoals present. These subgoals include 
Saving money and time, and to ensure design soundness, requirements, 
and equipment system compatibility. Additional goals include the veri- 
fication of publications such as system operating and maintenance pro- 
cedures, and system computer programs. Finally, it is a goal of the 
LBTS to provide the means to evaluate equipment performance and to 
solve system technical problems before they develop in the fleet and 
when they do, to provide a system to investigate and resolve them 


quickly. 


E. THE LAND-BASED TEST SITE CONTRACT 

Sperry Systems Management, an operating unit of Sperry Division, 
Sperry Rand Corporation, presently holds a $12 million dollar contract 
with Naval Ship Systems Command (NAVSHIPS) covering their effort in 
support of the PFCSLBTS. Some highlights of that contract (#N0024- 
73-C-1089) follow. 

1. Test and Evaluation 

Sperry Management Division is required to design, erect, and 

operate the land-based test site. The LBTS is to be used as a facility 


for integration and validation of the PF combat system prior to the 


10 l { x 





PF production decision. The Mk 92 MOD 2 fire control system being 
produced by the contractor under separate contract is the principal 
element of the combat system. The CSLBTS will be used for acceptance 
evaluation of the Mk 92 FCS and its subsequent integration with the GFE 
elements, simulators, switchboards and ancillary equipments of the 
PFCS for test validation and certification of the PFCS. 
2. Construction of the LBTS 

The LBTS is to include training facilities and the contractor 
shall define and provide system simulators and programs required, 
prepare equipment eee specification, statements of work and all 
other contractual documents necessary for procurement of the LBTS 
construction. The contractor will check out simulation equipment and 
programming to ensure adequate performance to tactical equipment's 
specifications. 

3. Support activities 

Under direction of the Navy OIC of the LBTS, the contractor 
is to provide a technical director to operate and maintain the LBTS 
and PCA, as a development demonstration proofing site for the PFCS. 
The contractor shall provide all facilities required by the OIC for the 
LBTS to include provision for and maintenance of the office space 
required by the OIC, provision for reproduction services, provision 
for clerical and custodial services, provision and maintenance of 


appropriate security services and precautions. 
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4. Repair parts 
The contractor will be provided spare and repair parts as 
determined by the government for government furnished equipment. 
5. PFCS Integration and Test 
The contractor shall integrate the Mk 92 MOD z FCS produced 
by the contractor under separate contract with government furnished 
computers, display, console sensor and other elements of the combat 
system at the LBTS. The contractor is required to verify the integra- 
tion of the combat system at the LBTS in accordance with a test plan 
to be provided by the contractor and approved by the Navy. The test 
plan shall be developed from an integrated test outline furnished by 
the Navy and shall be directed toward the verifying of combat system 
performance including the PFCS performance specification provided 
byeene* Navy. The contractor shall provide all test maintenance, com- 
puter support personnel, and such simulation programs as are required 
for the performance of this task. The contractor shall also collect, 
compile and analyze all data and prepare reports as required. The 
contractor shall develop appropriate test plans for performance of the 
IOT&E demonstration and submit the test plans for SHAPM approval. 
6. LBTS Operation and Demonstration Support 
The contractor shall support Navy personnel as required in 
performing the IOT&E demonstration at the LBTS. The contractor 
shall support other demonstrations and evaluations conducted at the 


LBTS under direction of the OIC. The contractor shall staff and 
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operate the LBTS facility under supervision of the OIC after completion 
of the initial PFCS integration, testing, certification and IOT&E demon- 
stration as a test development and training center. The facility will be 
utilized to support the PFCS inall areas requiring the use of an opera- 
tional test bed. Area use anticipated for the LBTS includes: 

~Documentation development and verification 

-Combat system crew training for follow ships 

-Special demonstrations 

-Investigation of combat system changes 

-Prove out of ordalts and shipalts 

- Fleet combat system problem investigation 

-Integration of growth items 

-Shipyard support. 

we. Lest PPS cedtl es 

Utilizing the integrated test outline and the combat system 
performance specification prepared by the Navy and furnished to the 
contractor, the contractor shall prepare and maintaina current test 
and evaluation plan, anda set of test procedures for use in the opera- 
tion, test, evaluation and demonstration of the PFCS at the LBTS and 
eventual use on the lead ship. The test plan and procedures shall cover 
the following: 

-Incoming inspection of GFE 


-Unit test of GFE 
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-Subsystem testing of GFE 
-Integration of the Mk 92 MOD 2 FCS produced by the contractor 
under another contract with material furnished under this con- 
tract and the GFE in the LBTS. 
-~Combat system test demonstration at the LBTS. 
8. SHAPM Support 
The contractor is required to support the SHAPM in the 
development, test and evaluation of the PFCS by maintaining control of 
the configuration of the PFCS at the LBTS until satisfactory completion 
of'the IOT&E demonstration. The contractor is required to furnish 
technical support to the lead ship builder in the design of the PFCS 
compartments. He is required to assist the SHAPM in the resolution 
of problems directly associated with the PFCS, specifically in the area 
of integration analysis, documentation, ship test planning and develop- 
ment, combat system reliability and maintainability and availability 
analysis and evaluation. 
The contract as outlined above expires in March of 1975 after 
the scheduled DSARC III presentation. It is conjectured that the con- 


tract will be extended on at least a yearly basis in support areas. 
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VII. SOME PFCSLBTS CONCEPTS AMPLIFIED 


A. GENERAL 

Contracts, test plans, operating procedures and Congressional 
testimony in generaladdress’ the theoretical side of procurement; 
what is planned or ought to be rather than what is. Since the Patrol 
Frigate LBTS can be viewed as a prototype of land-based test sites 
for future ship procurements, it is of interest to get behind the scenes 
to the grass roots level of test site concepts. Information presented 
in the following section ig an attempt to document the beliefs, gut 
feelings and thoughts of a myriad of individuals and commands contacted 
and/or visited during research for this thesis. The author and the 


author alone is responsible for the inferences contained herein. 


B. A COLLAGE OF QUESTIONS INCLUDING A FEW ANSWERS 

1. What working relationships have evolved between the LBTS 
OIC and SHAPM (PMS-399), Sperry NAVPRO etc. in the three month 
period the LBTS has been manned with Navy personnel? 

The OIC's marching orders at this early point in the LBTS life 
are essentially to get the job done. Since the concept of what the job is 
is evolving in parallel to the LBTS, tasks have not been formalized 
to any great extent, nor should they be. At the present time the OIC 


has broad duties and freedom in his actions. He has submitted a 
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one-page recommendation as to what his duties should entail, and 
PMS-399 is inthe process of integrating this input into a more formal 
job description. 

At the present time the OIC reports to NAVSHIPS as an OIC, 
and to PMS 399 as PMS 399.16, OIC of the PFCSLBTS. The NAVSHIP 
reporting chain enables the OIC to receive support such as being on the 
SNDL (Standard Navy Distribution List), have his own UIC (User Identi- 
fication Code), and enables him to receive outside assistance that could 
be cumbersome to receive via the SHAPM chain. The day-to-day chain 
of command is through PMS-399 and it is through this chain that fitness. - 
reports will be generated. | 

The OIC is administratively supported by the NAVPRO office 
at Sperry Great Neck, some 50 odd miles from the LBTS. This sup- 
port presently takes the form of Naval communication support, NAV-.... 
PRO PF/MK 92-2 contract inputs, and various other administrative 
support items. The NAVPRO is administering the Mk 92-2 contract 
with Sperry, but the PF CSLBTS contract is being administered by 
contracting personnel on the SHAPM's staff and by NAVSHIPS 025. 

The OIC's relationships with outside activities suchas NAVSEC, 
OPTEVFOR etc., generally falls in the area of assisting Sperry Man- 
agement Division in the prosecution of their contract. Informally, the 
OIC assists whomever he can. For instance, assistance is being pro- 
vided to NAVORD personnel in tracking NAVORD's portion of the Mk 92 


procurement contract. 
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The OIC is ona razor's edge between those who recognize 
a need in the fleet for a ship weapons system such as the PF and those 
who are test oriented, and at the extreme test for testing's sake. The 
former group are equipment/production oriented and include both pro- 
ducers of combat system equipment such as Sperry and operational 
personnel who perceive that the system, albeit with short falls, is 
better than what is presently inthe fleet. The latter group, the "testers", 
many times have to devote much time supporting the rationale of T&E 


reports before GAO, Congress, and the Office of the Secretary of 


Defensé to prevent misinterpretation. The reporting of T&E results ne ~. n,.~: 


culminates the T&E cycle and forms the basis for various decisions 
concerning the future status of the weapon system. For example, 
IOT&E reports are used by Congress as set forth in PL 92-156, and 

by the DSARC as an input for the major production decision. They are. 
also used by COMOPTEVFOR to support the suitability of a proposed 
system to meet fleet needs and operability requirements. 

In view of the importance of T&E, different agencies have 
different emphases in the conduct of the T&E. A grouping of the 
thrust of T&E needs could be the cost-effectiveness group versus 
operators and producers. Within the operator/equipment side, the 
LBTS OIC as stated in Chapter V has authority to make changes. 
Once the test group presents the OIC with a test plan for IOT&E and 
DSARC III demonstration, the OIC becomes an instrument of the test 


- group and his ability to alter plans is severely limited. 
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2. What authority does the OIC have within the boundaries of the 
contract? What is his relationship to testing and test and evaluation 
activities ? 

Authority to make changes in contractual obligations is 
severely limited, as it should be, in that the OIC does not have the 
resources to carry out complex changes. The OIC can make changes 
that involve no change in the scope of the contract and upon which Navy 
direction is required. PMS-399 maintains centralized authority to 
make contract changes. 

Since the effort required to derive proper tests and test pro-. 
cedures was underway long before the OIC reported to the LBTS in 
January 1974, the OIC had no input into test development. He has 
however, become deeply involved in the evaluation of the CSOT (Combat 
System Operability Test), which will be conducted by Navy personnel __ . | 
both at the LBTS and throughout the life cycle of projected ships. 

The OIC views testing at the LBTS as a three-phase effort 
as follows: | 

-Tests leading up to Navy acceptance/certification of the 
LBTS in December 1974. Sperry is conducting these tests based on 
outlines supplied by the Navy. The purpose of these tests is to demon- 
strate a total integrated combat system to the Navy as witnessed by 


the OIC, 
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-lOT&E tests. The purpose of these tests is a demonstration 
by the Navy to DOD and DSARC III of the operational characteristics of 
the combat system. The tests will include scenarios, simulated air- 
craft, and actual aircraft engagement. The OIC and Navy personnel 
will conduct these tests. The tests will be developed by PMS-399, 
APL (Applied Physics Laboratory), OPTEVFOR, and other agencies. 

-Prove out and conduct lead ship tests. These tests, from 
the Total Ship Test Program (TSTP), will be conducted to verify that 
the lead ship can indeed be checked out and delivered in an up condition. 
These tests will be conducted by the OIC and Navy personnel. ES 

At various stages inthe test phasing outlined above, different 
agencies will be involved in the actual data collection, data reduction, 
and analysis. These responsibilities are presently being considered. 

3.- ‘What formal and informal relationship exists between the.OIC.,....., 
of the LBTS and other NAVPRO personnel? coals 

There is no formal relationship such as one would find on an 
organizational chart. The OIC is working with the NAVPRO toward 
acceptance of the PFCS by the U. S. Government. There is presently 
under preparation an agreement between NAVSHIPS and the Sperry 
NAVPRO whereby the NAVPRO would be tasked to assist the OIC in 
all technical matters relating to the LBTS. | 

4. What is the status of follow-on contracts with Sperry Manage- 


ment for such things as life cycle support of the program? 
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Review of the present contract and the substance of future 
contracts will come under scrutiny in the fall of 1974. Unlike most 
contracts, it is envisioned that the contract for life cycle support of 
the program will be a multiple year contract. 

The LBTS is viewed as a ship permanently home ported in 
Sperry's building at MacArthur which the Navy runs. Sperry and other 
industrial support is viewed much like what afloat units receive from 
tenders and ship yards. 

5. How will personnel be cycled through the LBTS for training? 

“A solid long range plan has not been developed at this point.in 
time. Up through IOT&E, there will be a crew of about 22 men includ- 
ing three officers and one Wave resident at the LBTS. These people 
will constitute the Navy crew for the DSARC III demonstration, and 
then will eventually be assigned to the lead and follow ships. In parallel, 
additional people will cycle through the LBTS for training. 

About six to twelve sailors will be permanent residents of 
the LBTS under one officer in the grade of LCDR/CDR. They will 
operate and maintain the site and cycle through changes etc. as 
meguired. 

It is planned to instruct civilian workers from the three 
builders who win the PF procurement contract in all aspects of the 
installation and tests required by the PFCS. These individuals in 


turn will supervise installation and test of the PFCS at their respective 





building sites and instruct Navy precommissioning crews. These 
crews in turn will form the nucleus of the crew that takes the ship 
through trials and eventually to the fleet. 

6. What impact would stretching out of the production decision 
at DSARC III have on operation of the LBTS? 

The LBTS was primarily built as a vehicle to demonstrate 
the PFCS at DSARC III, and inthe long term, to support the life 
cycle of the ship. Inthis regard, at least in the short run, activity 
at the LBTS is directed toward the DSARC III presentation. These 
activities of themselves should be the cause for any delay of the produc- 
tion decision at DSARC III, rather than DSARC III causing the stretch- 
ing out. Other than a complete cancellation of the PF program then, 
the LBTS is here to stay. 

7.--- What is considered to be the ''salvage value" of the test site? . _ 

Assuming that the program receives a green light at the pro-., . 
duction decision, a realistic salvage value cannot be assigned to the 
Site at this time. That can only be really determined when the site 
is closed down. 

The major purpose of the LBTS is to determine if it is feasible 
to do what you want todo. In effect money is being spent now so that 
fifty times as much isn't required in the future to obtain the same 
results, whether good or bad. The $12 million or so invested in the 
LBTS is certainly a small cost in comparison with a projected cost 


of over $3 billion for the fifty ship buy. 





8. How closely does the LBTS duplicate both human and machine 


operations of the combat system aboard ship? 


It is the intent to duplicate as closely as possible shipboard 
conditions. Equipment limitations and the use of simulation was 
addressed in Chapter 5. From the human aspect, the LBTS is almost 
identical to a shipboard installation down to fire extinguishers on the 
bulkheads and Playmates posted in switchboard doors. 

Environmental factors preclude bringing jet aircraft against the 
site at high speeds and low altitudes, and simulators will have to be 
relied upon heavily for such tests. However, real life tends to be not ,.., 
as confusing, mixed up, nor saturated as the situations presented via 
simulations. In simulations, the kitchen sink can be thrown in but 
realistically, there are only so many aircraft that can come at you, 
and they are as confused as you are as to what the situation is. 

9. What will be the training status of the crew used for the 
DSARC III demonstration? 

For the PFCS, this question lends itself naturally to another 
one. How cana combat system be tested under ideal conditions when 
the training period leading up to the test period is minimal? Are you 
testing the combat system or are you testing the training of the demon- 
stration crew? 

At the LBTS, the Navy crew will have approximately one month 
"hands on'' training prior to the start of IOT&E. In this short time 


frame, they will be required to reach a level of competency often not 
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reached by fleet units during a six month buildup and several weeks of 
refresher training. 

Additionally, there exists an educational/technical gap between 
Navy and contractor T&E personnel. The contractor is represented by 
well-paid, highly experienced, and technically competent engineers and 
technical support personnel, which gives him the edge in structuring 
the T&E effort to his contractual advantage. 

10. How is approval for service use as a result of DSARC III, and 
the Board of Inspection and Survey (BIS) acceptance for service use 
from acceptance trials viewed by LBTS personnel? .- oo ner 

Sperry and LBTS Navy personnel understand that they are 
assembling a combat system to accomplish the mission required. Their 
effort is directed toward making the beast work. The BIS in its accept- 
ance trials tests not only how well the combat system performs its. . 
assigned tasks, but brings in points of contention suchas whethera .... .- 
certain light should be red or amber, are phone jacks located in acces- 
sible places (even though they may have been located in accordance with 
MILSPECS), and other such considerations. 

Admiral Zumwalt, testifying before a hearing on Cost Escala- 
tion in Defense Procurement Contracts in 1973, stated that he had 
command of a destroyer in 1957 which was one of 100 consecutive 
destroyers found to be unsatisfactory by the BIS. ''Yet my ship won 


the efficiency pennant in every department of the ship and the battle 
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efficiency pennant and yet if you read the INSURV report, we couldn't 
go to war."' 

The best way to handle deficiencies is to make sure they never 
arise inthe first place because of design or fabrication shortfalls. Thus 
it appears that there should be a closer relationship between BIS per- 
sonnel charged with acceptance of systems and personnel involved in 
presenting a system to the DSARC for service approval. 

11. How flexible is the site for expansion of PF projects? For 
alteration to similar ship systems? 

““"— The PMU is a physical model of the actual PF shipboard»: a: +n, 
spaces. As such it is a very useful tool to aid in determining the space 
allocation requirements for growth items. The LBTS itself is space 


limited to a certain degree within the existing building if it is desired 


to*build:PMU's of other classes of ships. However space does exist,, 1. 


whereby equipment could be placed on pallets and connected to the 
PCA to conduct various equipment/computer interface tests. Real 
estate wise, grounds exist to extend the existing building by a con- 
siderable amount. 

12. What systems will be taken from the LBTS to Bath for instal- 
lation aboard the lead ship? Will specific subsystems be tested/ 
integrated at the LBTS prior to being sent to the shipbuilders? 

There is a possibility that all the equipment comprising the 


PMU installation will be sent to Bath for installation aboard the lead 
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ship. In this manner, shipboard installation and check out would have 
as its basis a completely checked out combat system. Presently under 
consideration is what specific subsystems, if any, should be cycled 
through the LBTS prior to shipboard installation. Evena partial 
cycling of critical units through the LBTS would improve configuration 
control, enhance check-out of the equipment, and provide an opportunity 
for LBTS personnel to bring equipment up to the latest change status 
and conduct system level tests on the equipment integrated into the 
system. ‘ 

“Contractors and shipbuilders are instructed to install GFE jin. .., 
ships. Often the equipment to be installed has not been adequately 
modified to reflect the latest change status, or parts are missing. 
The LBTS has not escaped this either. For instance, the CWI units 
for the Mk 92 FCS were received without several cards and/or drawer __ 
components. Often such equipment is installed and equipment/sub- .. 
system level tests are in progress before it is discovered that a crystal 
buried in the back of a chassis is missing. Cycling high risk items 
through the LBTS would help to alleviate this problem. 

Another concept that merits consideration encompasses the 
cycling of equipment through the LBTS where it could be checked out 
by a verification and test team. This same team could follow the sys- 
tem on board ship where they would be responsible for integration 


testing. Such an approach would definitely reduce test time and since 
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the follow ships are scheduled to go down the ways at the rate of one 
per month, it is feasible from a manning standpoint. 
13. How were the following site selection criteria decided? 
-Cost effectiveness (stressing maximum utilization of existing 
assets consistent with the mission of the facility employed)? 

Once Sperry received the contract to design, build and operate 
the PFCSLBTS, the decision as to where the test site should be located 
was Sperry's. Thus cost effectiveness has to be viewed from Sperry's 
viewpoint and this information is not available. 

-Minimum MILCON requirement. . ae ae 

It is understood that one of the factors leading to the decision 
to place Sperry under contractual obligation to build the test site with 
R&D funds was the long lead time required to obtain funding via MIL- 
CON channels. MILCON funding requirements were thus minimized ___. 
ero): but whether or not a greater amount of R&D funds were required 
to erect the LBTS is unknown. 

-Adequate source of professional talent and skilled labor, good 
housing, and community services. 

Once again such considerations were made by Sperry. It is 
understood that considerations such as those listed above came to bear 
in Sperry's decision, but again, this information is not available. 

-Availability of fleet computer programming groups. 
It does not appear that this was a consideration. UNIVAC is 


tasked with the development of programs, and Sperry with the running 
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and check out of these programs. While Navy fleet computer groups 
are monitoring the progress and will become actively involved in the 
certification of the PFCS, it does not appear that their availability was 
a factor in site selection, 

-Availability of fleet units in the area. 

This was not a consideration as can best be determined. 

~Availability of controlled Navy aircraft services with minimum 
airspace restriction. 

Sperry has previously used the area in which the PFCSLBTS is 
located tO conduct tests on the AN/SPG-55B Terrier FCS. Based-on .- 
this experience it was postulated that adequate aircraft services could 
be obtained to support PFCS tests. It is obvious that more realistic 
flight paths could be programmed if the LBTS were remotely located, 
and this-is another tradeoff that was made in site selection. 

~Easy accessibility coupled with good security. .- 

Again this was a Sperry decision. Under the terms of their 
contract Sperry is responsible for providing security at the LBTS. 

No fault can be found with their system which consists of a badge and 
sign in/out log. Visitors must be signed in by site personnel and 
escorted. The site itself is easy to reach and more than adequate 
parking is provided. 

~Near large body of water (for electromagnetic propagation 
considerations). 


There is no problem in this area on Long Island. 
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-Controlled electromagnetic environment. 

Sperry conducted many ECM/ECCM checks on the AN/SPG- 
55B FCS. While the environment around the LBTS can be controlled 
only to a limited extent, it must be remembered that the radars asso- 
ciated with fire control systems operate ina frequency range not 
normally employed by civilian activities. As such then, little inter- 
ference with FCS ECM/ECCM testing is anticipated. 

-Accessibility to collocated NAVSEC engineering groups and 
SHAPM. | 
~~" Unless the LBTS was located in Washington, the present 
location is as good as might be expected. It is located a stone's throw 
from MacArthur Airport and airline schedules to and from Washington 
are such that people coming to the LBTS from Washington can arrive 
‘in-the*morning and depart in the afternoon with a minimum of delay. ..,,, 

14... How will the test site be certified? What does certification. 
of the test site mean? Are the simulations being used certified? 

fhe LBIis will be certified by a certification activity desig- 
nated by PMS-399. At the present time (March 1974) it looks like this 
activity will be the eastern detachment of the Naval Ships Missile 
System Engineering Station. 

Certification, as defined within PF Programs, is the process 
of conducting a test procedure at the PFCSLBTS (or other facility) in 


order to ensure that it is correct, accurate and complete with respect 
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to the equipment being tested. A test procedure will be certified 

as being correct on satisfactory completion of certification. Certifica- 
tion of the PFCSLBTS in essence means that the site has been tested 
as a combat system and the results of these tests show that the site 
does indeed operate as the combat system designed. 

Simulations being used at the LBTS at the present time do not 
have a certification in the sense of acceptance via testing in accordance 
with an approved test plan. ‘Sperry is developing a plan to demonstrate 
that in fact the simulations represent actual equipment. In the sense 
that individual simulations support the overall conduction of certifica- 
tion tests, the simulators will be certified at the time the PF CSLBTS 
is certified. 

15. What factors will be introduced during testing to simulate 
power fluctuations, power failures, heat and cold, air conditioning 
failures, chill water that isn't chilled, etc. ? 

None. Environmental testing to required specifications is to 
have been completed during equipment manufacture and test. Deliberate 
failures are not planned to be introduced during the course of testing at 
the LBTS. The site is powered from the Long Island Power Company, 
and this power is thought to be less regulated and more apt to drop 
than shipboard power. There will probably be a sizeable number of 


other type failures encountered in the normal course of events. 
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In summary, this section attempts to pose some pointed questions 
concerning the PFCSLBTS, and to provide some answers. Life being 
what it is, there always appears to be more questions than answers. 
Much was left unquestioned and unanswered suchas: ''What is the 
value and what has been said and done about the necessity of developing 
and retaining, permanently, within the Navy, the expertise required to 
integrate a combat system?'"'; ''How will the LBTS be integrated with | 
BIS and OPTEVFOR for lead and follow ship testing? ' and ‘What are 


the real cost/effectiveness considerations for the LBTS?'"! 
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VIII. CONCLUSION AND AREAS FOR FURTHER CONSIDERATION 


A. GENERAL 

This thesis should be considered neither a compendium of combat 
system test and evaluation nor a handbook of solutions based on the 
approach of the Patrol Frigate program via a land-based test site. 
It does trace emerging doctrines encompassing ship acquisition T&E 
and investigates the PF LBTS as an innovative approach to a practical 
solution. The complexities of modern shipboard combat systems have 
been discussed as has efforts to formalize the test and evaluation 
requirements for these systems. To date however, the strides that 
have been made in T&E are due largely to the confiction of the managers 
involved. The use of land-based sites in T&E investigated in this thesis 
has been developed from an empirical rather than a theoretical analysis. 
Successful implementation of T&E policy must be based primarily on 
past experience, and personnel involved with ship T&E policy should 
have some background in ship acquisition programs both from a the- 
oretical anda "hands on" base. This will allow thorough attention to 
proposals for future improvements that will continually be present in 


this emergent field. 


Ee owe TION AT THE PrELBTS 
In recognition of the cost to procure, install, operate, and main- 


tain a total ship combat system, including all its sensor and weapons 
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subsystems, the PFLBTS reles on simulation for interface testing of 
certain hardware and software items. This imposes limitations on the 
facility; these limitations appear acceptable in that the PFLBTS has 
consistently retained the option, within funding constraints, for instal- 
lation of further equipment and subsystems should experience prove it 
to be vital to the T&E program. 

Simulations provide T&E personnel the opportunity to subject men 
and machines to a much wider variety of test situations than one could 


normally expect when working with complete systems. 


C. LBTS COST/EFFECTIVENESS CONSIDERATIONS 

When considering the expenditure of funds required for the estab- 
lishment and operation of a LBTS, it should be remembered that, with 
few exceptions, these costs will be incurred anyway, i.e., if not at the 
LBTS, then somewhere else. The task of proving the integration of the 
total ship combat system must necessarily be Ne perae Hed. 

Although the "effectiveness’' of a test facility is not amenable to 
precise definition or measurement, its cost can be defined and es- 
timated with some certainty. If alternatives in execution of a TSTP 
are limited to T&E ata LBTS or T&E aboard ship, then both alter- 
natives could by definition be "effective'' and thus have a constant 
and equal effectiveness, i.e., they are of "equal benefit'' in economic 


analysis terms. 
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As previously stated, the costs of the LBTS itself can be deter- 
mined with some degree of precision. The cost of obtaining a success- 
ful T&E via fragmented testing ata diversity of test sites under 
different managers may only be conjured up. Based on the experiences 
of past T&E programs such as the DLGN-36, it is postulated that ship- 
Beara testing, having as its basis adequate LBTS support, is much less 


costly. The answer is locked in the future. 


D. ADVANTAGES OF LAND.BASED TESTING 

Some advantages of utilization of a dedicated LBTS such as that 
being employed in the PF Program include: 

1. Software debug and testing can be accomplished ina controlled 
hardware environment. 

2. System integration debugging can be started earlier since it 
is not tied to the ship construction schedule and, parenthetically, con- 
struction delays. 

3. Software debugging can be accomplished in an environment 
free of industrial work. 

4. Use of simulation and simulators can be optimized both ona 
single project and multi-project levels. One simulator can support 
more than one system integration effort as can hardware. 

5. A single integration facility eliminates redundant test 


equipment. 
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6. From the ''people" perspective it concentrates your best 
people at one site. These are the same people that follow the system 
integration effort through to final ship acceptance. 

7. Support software, debug aids, data extraction and reduction 
and analytical tools can be provided on-site and shared by more than 
one project. 

8. Test site provides earlier "hands on" experience for the 
nucleus crew and key personnel. 

9. GFE software can be "certified" prior to delivery to the 
contractor shipbuilder. 

10. The test site is a step toward satisfying the necessity to 
develop and retain, permanently, within the Navy, the expertise 
required to test, evaluate, and integrate a ship combat system. 


ll. It meets DOD requirements to 'fly-before-buy." 


E. T&RE AND THE PATROL FRIGATE 

Admiral Woodfin and others have cited several reasons why the 
PFCS integration should not proceed down the ''primrose path" taken 
by previous ship acquisitions. The PF Program has integrated total 
ship testing into the design from the outset. This includes HME, 
combat system, and mobility. The integrated test program includes: 
receipt inspection, installation and check out of equipment, equipment 
testing, intra system tests, inter system tests, builder trials, and 


acceptance trials. Integration includes utilization of a dedicated 


125 ao 





CSLBTS that is a faithful reproduction of the ship. The computer soft- 
ware, combat system integration, and test site development is being 
performed by one contractor. The tests performed at the test site 
will be directly transferable to the ship and test site personnel will be 
involved in the shipboard testing. The PF Program has . combat sys- 
tems integration manager, responsible for the total system. Finally, 
the lead ship is being acquired ona cost type contract basis, which 
will facilitate dealing with integration problems as they occur. 

In conclusion, although there is little in major weapon system 
acquisition that can be predicted with absolute certainty, upon careful 
consideration, few can deny the legitimacy and fertility of the utiliza- 
tion of a LBTS to reduce uncertainty in ship acquisition testing. ''By 


your test results, ye shall be known", could well serve as a motto for 


these nCSLE TS. 


F, AREAS FOR FURTHER CONSIDERATION 

1. During the time frame that this thesis was being researched 
and written, the PFCSLBTS was in the process of being built. All of 
the equipment was not scheduled to have arrived, and the bulk of the 
equipment at the site was in the process of being powered. Subsystem 
and system testing obviously had not started. It would be of interest 
to continue to follow the evolution of the PFCSLBTS through to its 


final wake, be it at the end of DSARC III, or when the last PF is 
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decommissioned some thirty years or so hence. Main points for 
consideration include: 

.Did the PFCSLBTS fill its mission to provide the facilities and 
personnel to test integrated ship combat systems. 

-How effective was the LBTS in discovering and correcting (or 
causing correction), subsystem, interface and computer program 
problems. 

-~What was the post audit cost of the test site and did indeed a 
constant effectiveness situation exist between the LBTS and other 
viable alternatives. 

2. The use of a LBTS and of prototyping has been proposed as an 
alternative conducive to reducing uncertainty in an acquisition program. 
A subjective analysis of available alternatives leads one to the con- 
clusion that a LBTS is a viable approach. Would the same results 
be obtained from a quantitative analysis? 

3. Gordon W. Rule ina recent newspaper article, advocated 
slowing the production schedule of the PF to gain additional time for 
testing of the Mk 92 FCS, a Dutch designed system, and the Mk 75 gun, 
an Italian design. Another article reported that the cost of the PF is 
now about $70 million each, compared with $45 millionin 1971. It 
would be of interest to analyze actual costs with budgeted costs of the 


Pil Propram, tempered by inputs suchas Mr, Rule's. 
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APPENDIX A 
TEST MODEL FOR ACQUISITION CYCLE 
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